Performance of Two Photoscreeners Enhanced by Protective Cases.

BACKGROUND: Photoscreeners provide valid, cost-effective early detection of amblyopia risk factors; however, they require proper illumination, flash intensity, pupil dilation and patient cooperation. The Kaleidos case for the 2WIN infrared photoscreener fixes focal distance, decreases luminance and provides electric power. GoCheck Kids (GCK) has developed a flash-concentrating case for the Apple iPhone 7 Plus smartphone to reduce exposure time.
METHODS: In remote Burmese and urban Alaskan clinics, Kaleidos and GCK were used before a confirmatory eye examination using 2013 AAPOS uniform guidelines validation, including some older patients. 2WIN refraction was compared to a cycloplegic examination using J0 and J45 vector transformation.
RESULTS: In total, 48 Burmese patients and 114 Alaskan patients aged 8±6 years were evaluated utilizing 2013 amblyopia risk factor guidelines. Kaleidos, with 13 Burmese inconclusives, had a sensitivity of 84%, specificity of 65% and positive predictive value (PPV) of 63%. GCK, with only six inconclusives, had a sensitivity of 63%, specificity of 83% and PPV of 73%, with sensitivity improving to 74% with central expert interpretation. Kaleidos closely matched cycloplegic refraction, with intraclass correlations of 0.47 for J0 vector and 0.57 for J45 vector. The protective cases provided clearer images and better pupil dilation than similar devices without cases.
CONCLUSION: Both devices detected amblyopia risk factors well. GCK gave fewer inconclusive results even with the Asian eyelid configuration, while Kaleidos matched sphere and cylinder refraction. The specialized housing enhanced the performance of the 2WIN and GCK photoscreeners, improving the speed and reliability of amblyopia screening and refraction, even in populous and luminous locations. CLINICAL TRIALS REGISTRY: NCT04068129.

Introduction

Photoscreening is an objective technique for detecting amblyopia risk factors early enough to reduce permanent vision impairment.1–4 Amblyopia is deficient acquisition of central nervous system vision during a critical period of a child’s first decade.5 Risk factors for amblyopia include obscured images (ie infantile cataract), constant strabismus and refractive error. Effective treatment of amblyopia includes early reversal of the offending risk factors and directing visual influences to the amblyopic eye. The vision screening committee of the American Association for Pediatric Ophthalmology and Strabismus (AAPOS) has published two related sets of uniform guidelines for the comparative evaluation of cost-effective screening for target levels of the main amblyopia risk factors, the first in 2003,6 followed by the 2013 age-based update.7

Photoscreeners work by shining visible or infrared light at both eyes and then analyzing the pupillary red reflexes. Current instruments can provide instant computer determination as to whether the patient passes or should be referred for confirmatory pediatric ophthalmic examination.8 A dimly lit environment, with children who are cooperative, sit still and have large pupils are the optimal conditions for producing reliable results with photoscreeners. Challenges arise when dealing with patients who cannot look at the fixation targets on the devices, or when their pupils are too small or partially obstructed by eyelids or eyelashes, which has been a notorious photoscreening challenge in Asian children. Two photoscreeners that have been enhanced with special housing to address some of these challenges of pediatric vision screening worldwide are the computer-interpreted Kaleidos for the 2WIN (Adaptica, Padova, Italy) and smartphone-interpreted GoCheck Kids (GCK, aka Gobiquity, Scottsdale, AZ).

2WIN is a computer-interpreted infrared photoscreener with similarly high validity when compared to two other infrared photoscreeners: Plusoptix (Nuremberg, Germany) and SPOT (Welch-Allyn, Skaneateles Falls, NY).9 2WIN was developed by Mario Angi as a component of a portable suite of refractive tools for adults, and also for children. 2WIN refractometry resembles the high quality of Retinomax (Righton, Tokyo Japan).10 A modified housing for 2WIN called Kaleidos has been developed; this is a rectangular tube which, on one end, clamps the 2WIN with additional battery power and tablet remote control, while the other end of the housing provides a darkened tunnel with the desired focal distance that encourages the patient to gaze undistracted at the 2WIN fixation lights, while also naturally dilating the pupils.

In 2013, we discovered that an LED-flash iPhone (Apple, Cupertino, CA) smartphone can provide valid photoscreening if the child sits still.11 GCK then provided interpretive software adapted to evolving models of smartphones allowing cost-effective photoscreening. The actual flash of the Nokia Lumia 1020 (Espoo, Finland) was initially cropped and utilized, with excellent positive predictive value (PPV).12,13 For typical American children showing astigmatism, predominantly, with-the-rule, one-axis exposure had similar validity to slower, two-axis exposure.14 The current GCK app uses an iPhone with an LED source which requires longer than instant exposure to gain sufficient pupillary red reflex.15 GCK has developed an enhanced housing for the iPhone 7 Plus that concentrates the flash by adding a convex lens centered over the LED “flash”, speeding up the process, reducing blur and facilitating two-axis photoscreening. Smartphone software was then adjusted to account for the brighter, more concentrated red-reflex illumination.

We compared the 2WIN in the Kaleidos system and the GCK flash-concentrated housing in a remote clinic in Burma, and also in an Alaskan urban pediatric ophthalmology clinic.

Methods

This reliability analysis was covered by the Institutional Review Board at Providence Hospital, Anchorage, Alaska ( Clinical Trials Registry no. NCT04068129). Depending on age, the patients, parents and/or legal guardians provided written informed consent or assent. For Burmese patients, risks and benefits were translated into the native S’Gaw or Po Karen language. The study complies with HIPAA and the Declaration of Helsinki. Patients received medical care but no stipend was paid for research participation. This study of screening tests complies with STARD 2015.

Photoscreening with follow-up confirmatory examinations for referred and passed interpretations was offered to children and young adults at a remote clinic in the Karen State of eastern Burma (Myanmar). The clinic had an intermittent 220-volt power generator, but no internet coverage.

Consecutive patients in a WiFi-equipped pediatric ophthalmology practice in Anchorage, Alaska, underwent photoscreening before confirmatory examination.

Each patient underwent photoscreening with the 2WIN photoscreener installed in the Kaleidos housing. In Anchorage, the device was controlled by the accompanying wireless tablet computer. In Burma, however, despite using a WiFi router not connected to the internet, the tablet computer would not connect to the 2WIN; therefore, the rear trapdoor of the Kaleidos was opened so that the 2WIN could be activated and controlled manually (Figure 1). The 2WIN stored results on a micro-SD memory card, which was eventually downloaded to a computer.

Figure 1

Kaleidos enhanced housing for 2WIN photoscreening, with housing opened to allow manual control. (Parental permission given for use of all photographs).

Each patient also underwent photoscreening with GCK on an iPhone 7 Plus using the enhanced, flash-concentrating cell-phone case. Pupil size and ocular media were viewed in real time after successful image acquisition. In Anchorage, in addition to on-site smartphone interpretation, images were uploaded to the manufacturer’s central reading center for secondary interpretation. GCK as a photoscreener therefore has the unique option to provide on-site smartphone interpretation and also internet-transmitted, expert reinterpretation by the manufacturer; both results were analyzed. In Burma, all images were retained on the smartphone and eventually uploaded for central secondary analysis by GCK after returning to urban internet availability. In Burma, a portable, lightweight tent was used to provide a dim screening environment with less peripheral visual distraction (Figure 2).16 The order of Kaleidos and GCK photoscreening was randomized and the confirmatory examiner was not aware of the screening results at that time.

Figure 2

GoCheck Kids iPhone 7 Plus with flash-concentrating case being used in a tent in Burma.

Confirmatory examinations were performed following 2013 AAPOS uniform guidelines, modified so that older patients were grouped with the older triad of patients (“2013+” AAPOS). Additional analyses were performed with the 2003 guidelines and patients with amblyopia. Amblyopia is defined as visual acuity of 20/40 or worse and/or two-line intereye difference combined with amblyopia risk factors. Cycloplegic refractions were performed 30–40 minutes after instillation of cyclopentolate 1%. Visual acuity was checked with patched, surround HOTV at 3 meters, at a threshold of at least three out of four correct optotypes. Validation was performed with a conventional 2×2 screen-examination matrix, and also with an ABCD 2×3 matrix, which incorporates inconclusive results (no instrument interpretation or unable to gain a reading) which are considered as referrals (Table 1).17 No patients were lost to follow-up.

Table 1

Validation of Kaleidos and GoCheck Kids Compared to Age-Added 2013 Amblyopia Risk Factors and True Amblyopia

	x+	x–
s+	A	B
s–	C	D
si	E	F	n=162				2×2					i=r
2013 ARF	A	B	C	D	E	F	sens	spec	PPV	NPV	Y J	sens	spec	PPV	NPV	Y J
Kaleidos	55	19	11	61	2	14	83%	76%	74%	85%	60%	84%	65%	63%	85%	49%
GCK	40	13	25	78	3	3	62%	86%	75%	76%	47%	63%	83%	73%	76%	46%
G-read	47	30	18	61	3	3	72%	67%	61%	77%	39%	74%	65%	60%	77%	38%
Amblyopia	A	B	C	D	E	F	sens	spec	PPV	NPV	Y J	sens	spec	PPV	NPV	Y J
Kaleidos	33	41	2	70	1	15	94%	63%	45%	97%	57%	94%	56%	38%	97%	50%
GCK	31	23	4	98	1	5	89%	81%	57%	96%	70%	89%	78%	53%	96%	67%
G-read	28	49	8	71	1	5	78%	59%	36%	90%	37%	78%	57%	35%	90%	35%

Notes: Kaleidos is the Adaptica 2WIN infrared photoscreener in a protective case while GoCheck Kids is an iPhone 7 Plus in a flash-concentrating protective case with local interpretation (GCK) as well as central expert reading interpretation (G-read). A–D are values for screen “refer” (s+) and screen “pass” (s–) compared to confirmatory examination risk factors present (x+) and examination risk factors not present (x–). Values E and F are when the screening device was inconclusive or unable to deliver a definitive result (si). Examination outcomes are classified as actual “amblyopia” or amblyopia risk factors stratified by age defined by AAPOS in 2013 (2103+ ARF) with the older age triad expanded to include all subjects older than 48 months. Sensitivity (sens), specificity (spec), positive predictive value (PPV), negative predictive value (NPV) and Youden’s J (Y J) are presented regarding just the A–D conventional subset (“2×2”) as well as considering each inconclusive screening result (si) as if it were a refer (i=r).

Twenty of the Anchorage screenings were compared with prior versions of GCK (iPhone 7 Plus alone, Nokia Lumia 1020) and also a hand-held Adaptica 2WIN without a Kaleidos case.

The sample size for a correlation with type I error alpha of 0.01 and type II error beta of 0.05, expecting a correlation of r=0.6, should comprise at least 40 subjects.

Results

In Burma, 48 patients aged 15±8 years (mean±SD) completed testing with a prescreening probability of 2003 AAPOS risk factors of 8%. These were compared to and added to 114 patients from Alaska, with mean±SD age 7±4 years and prescreening probability of 60%. The median age for all 162 patients was 8.2 years. Only 50 subjects met the exact age ranges for 2013 AAPOS age divisions less than 72 months and therefore we added older patients to the older triad, defined as “2013+.”

Table 1 gives the performance of the Kaleidos and the GCK photoscreeners compared to age-enhanced 2013 AAPOS guidelines and also compared to actual amblyopia. Figure 3 shows receiver operating characteristics (ROC) curves for Kaleidos and flash-concentrated GCK compared to 2013+ AAPOS amblyopia risk factors and also to actual amblyopia, with any inconclusive results considered as a “refer.”

Figure 3

ROC curve. Performance of GoCheck Kids iPhone 7 Plus with flash-concentrating case (GCK, circles) and 2WIN photoscreener in Kaleidos enhancing case (Kaleidos, squares) validated against the age-enhanced 2013 AAPOS uniform amblyopia risk factor (ARF) guidelines (solid bold lines) and amblyopia (fine, dashed lines).

Validation is impacted by how inconclusive interpretations are considered,17 by instrument referral criteria, by prescreening probability and by gold-standard examination criteria. The shows validation metrics (sensitivity, specificity and PPV) for Kaleidos and GCK housing-modified photoscreeners separated by Burmese and Alaskan cohorts with the corresponding instrument referral criteria. By 2013+ AAPOS amblyopia risk factor uniform guidelines with inconclusives considered as “refer”, the sensitivity, specificity and PPV for Kaleidos in 48 Burmese patients were 75%, 68% and 18% and in 114 Alaskan patients 86%, 62% and 74%, respectively. In comparison, GCK on-site had respective values of 50%, 86% and 25% in Burmese and 64%, 80% and 80% in Alaskan patients, but central interpretation yielded 100%, 60% and 18% in Burmese and 72%, 68% and 74% in Alaskan patients, respectively. When inconclusive interpretations were incorporated as if they were referrals, the performance of Kaleidos suffered more than GCK. The sensitivity of GCK on-site interpretation was increased by remote, central expert reading.

Figures 4 and 5 show refraction estimates of Kaleidos by J0 and J45 power vectors, respectively, compared to retinoscopy. For all patients with cycloplegic refraction sphere hyperopia greater than 0.5 D, the mean±SD hyperopia was 2.1±2.1 D while the reading from the Kaleidos was 0.4±1.4 D. Figures 6 and 7 show corresponding Bland–Altman plots. Table 2 shows Bland–Altman metrics and intraclass correlation coefficients from refractive estimates for hyperopia, myopia and astigmatism degree, and J0 and J45 vectors, with GCK and Kaleidos compared to cycloplegic examination.

Figure 4

Kaleidos J0 vector analysis comparing Kaleidos refractions with cycloplegic retinoscopy.

Figure 5

J45 vector analysis comparing 2WIN photoscreening through Kaleidos protective case with uniform guidelines cycloplegic refraction.

Figure 6

Bland–Altman plot for J0 vectors comparing Kaleidos with retinoscopy.

Figure 7

Bland–Altman plot for J45 vectors comparing Kaleidos with retinoscopy.

Table 2

Correlations Between Kaleidos Refractions and Cycloplegic Examination

	Bland–Altman				Intraclass Correlation
	Mean Diff	Lower	Upper	St Diff	ICC	Low ICC	High ICC
GCK sphere	1.05	−2.34	4.45	1.73	0.66	0.41	0.79
GCK aniso	0.25	−1.21	1.71	0.74	0.62	0.46	0.73
GCK astig	0.33	−2.03	2.71	1.21	0.64	0.52	0.73
K sphere	0.41	−2.19	3.05	1.37	0.82	0.76	0.87
K aniso	0.06	−1.03	1.14	0.56	0.8	0.74	0.86
K astig	−0.15	−1.52	1.21	0.69	0.83	0.79	0.87
K-2WIN J0	−0.08	−0.76	0.6	0.35	0.47	0.38	0.55
K-2WIN J45	0	−0.5	0.5	0.25	0.57	0.49	0.64

Notes: GoCheck Kids with flash-concentrating case (GCK) and 2WIN photoscreener in Kaleidos case (K) comparing spherical equivalent (sphere), anisometropia (aniso) and cylinder (astig) against cycloplegic refraction. The Bland–Altman metrics include the mean difference (Mean Diff) with confidence limits and the standard deviation of the difference (St Diff). Correlations are demonstrated by intraclass correlation coefficients (ICC) with confidence limits. Kaleidos refractions were compared to cycloplegic refractions using the J0 (K-2WIN J0) and J45 (K-2WIN J45) vectors.

The practical ability to obtain focused photoscreen images and yield interpretation in 20 patients was compared with prior instruments lacking the new protective cases. In the daytime clinic in Burma and in the lit examination rooms in Alaska, Kaleidos was easily able to yield focused images with mydriatic pupils, while the hand-held 2WIN would not provide an interpretation owing to either “pupils too small” or excess infrared light levels. Opening the trapdoor back of Kaleidos in Burma produced adequate photoscreening images similar to tablet control in Anchorage, but activating the shutter was not simple. Some Burmese patients could not open their eyelids enough for the 2WIN to yield results with or without the Kaleidos case. In patients who did not easily sit still, prior GCK cameras required more repeat efforts and resulted in more blurred pupils compared to those images produced by the new flash-concentrating case. As with our past experience, the tent enhanced pupil dilation and fixation compared to middle-of-the-room screening efforts with GCK in the Burmese patients.

Conclusion

Protective containers improved illumination and fixation for the 2WIN photoscreener, and the flash-concentrating external case improved image quality by shortening exposure time with the GCK app on the iPhone 7 Plus. Kaleidos refractions closely resembled sphero-cylinder refraction, but underestimated cycloplegic hyperopia, as expected.18 GCK gave fewer inconclusive readings than 2WIN, especially in Burmese patients. Central expert interpretation after internet download to the manufacturer improved validity for sensitive GCK screening by a moderate amount.

Our experience with both devices with protective enhancing cases compared well with other photoscreening studies.3 When inconclusive results are counted as “refer”, in a cohort with 45% prescreening probability of 2003 AAPOS risk factors, Kaleidos had 74% sensitivity and 82% specificity while GCK had 78% sensitivity and 86% specificity. Bare 2WIN, with its CR strabismus tool but no protective case, had 68% sensitivity and 84% specificity in a cohort with 56% prescreening probability of 2003 AAPOS risk factors and 31% developmental delay.10 GCK with the Nokia 1020 smartphone had 65% sensitivity and 83% specificity on site, improving to 76% sensitivity and 85% specificity with expert central reading, in an enriched cohort with 38% prescreening probability.15 In another group with 36% 2013 prescreening probability, GCK Nokia 1020 had 76% sensitivity and 67% specificity on site, and also improved with central expert reading.12 GCK Nokia 1020 referred 5% of 6310 pediatrician screenings for a 2003 AAPOS PPV of 77%.13 In head-to-head screening of Plusoptix compared to SPOT in 270 patients with 63% prescreening probability, the sensitivity and specificity for Plusoptix were 84% and 94% and for SPOT were 82% and 68%, respectively,17 and again with 62 children, the sensitivity and specificity for Plusoptix S12 were 91% and 71%, for SPOT 78% and 59% and for 2WIN 73% and 76%, respectively.9

The prevalence of 2003 AAPOS amblyopia risk factors (prescreening probability) in American communities is about 21%.19,20 Community amblyopia screening excels when the PPV is 80% or greater21 and specificity exceeds 90%.22 Since the prevalence of AAPOS amblyopia risk factors exceeds the prevalence of amblyopia, 100% sensitivity is not preferred, and with early specific instrument referral criteria, a sensitivity near 50% may be adequate.20 By these standards, the protective-cased Kaleidos and GCK photoscreeners performed admirably.

Advantages of this study are the inclusion of remote and urban patient cohorts, one with low and one with high prescreening probability for refractive amblyopia risk factors according to the uniform guidelines. In community screening, strabismus is much less common than in pediatric eye practice; most novel cases of strabismic amblyopia are associated with apparent, large-angle constant strabismus, for which scheduled photoscreening is inferior to immediate parental referral for timely detection.

Weaknesses of this study are the relatively small number of subjects with ages represented by 2013 AAPOS guidelines screening. We also did not screen with and without the protective cases in all patients. When lacking an internet connection in Burma, the Kaleidos system seemed to perform well when the operator manually opened the clamshell, but we cannot be sure that this method truly mirrored the routine use in the Alaskan patients.

Adaptica could improve their results by working to better interpret dark irides with Asian eyelid and eyelash configuration, and by redesigning the camera shutter when internet and WiFi are not available for Kaleidos. GoCheck Kids could improve their screener by improving fixation and accommodation directed at their smartphone lens or the protective case.