Test-Retest Variability of Functional and Structural Parameters in Patients with Stargardt Disease Participating in the SAR422459 Gene Therapy Trial.

PURPOSE: The goal of this analysis was to determine the test-retest variability of functional and structural measures from a cohort of patients with advanced forms of Stargardt Disease (STGD) participating in the SAR422459 (NCT01367444) gene therapy clinical trial.
METHODS: Twenty-two participants, aged 24 to 66, diagnosed with advanced forms of STGD, with at least one pathogenic ABCA4 mutation on each chromosome participating in the SAR422459 (NCT01367444) gene therapy clinical trial, were screened over three visits within 3 weeks or less. Functional visual evaluations included: best-corrected visual acuity (BCVA) Early Treatment Diabetic Retinopathy Study (ETDRS) letter score, semiautomated kinetic perimetry (SKP) using isopters I4e, III4e, and V4e, hill of vision (HOV) calculated from static visual fields (SVF) by using a 184n point centrally condensed grid with the stimulus size V test target. Retinal structural changes such as central macular thickness and macular volume were assessed by spectral-domain optical coherence tomography (SD-OCT). Repeatability coefficients (RC) and 95% confidential intervals (CI) were calculated for each parameter using a hierarchical mixed-effects model and bootstrapping.
RESULTS: Criteria for statistically significant changes for various parameters were found to be the following: BCVA letter score (8 letters), SKP isopters I4e, III4e, and V4e (3478.85; 2488.02 and 2622.46 deg2, respectively), SVF full volume HOV (VTOT, 14.62 dB-sr), central macular thickness, and macular volume (4.27 μm and 0.15 mm3, respectively).
CONCLUSIONS: This analysis provides important information necessary to determine if significant changes are occurring in structural and functional assessments commonly used to measure disease progression in this cohort of patients with STGD. Moreover, this information is useful for future trials assessing safety and efficacy of treatments in STGD. TRANSLATIONAL RELEVANCE: Determination of variability of functional and structural measures in participants with advanced stages of the STGD is necessary to assess efficacy and safety in treatment trials involving STGD patients.

Introduction

Since mutations in the ABCA4 gene were shown to be causative of autosomal recessive Stargardt disease (STGD) by Alllikmets,[1,2] multiple publications have described the classification, general course, genotype–phenotype variability, and severity of STGD.[3-7] Mutations in ABCA4 are associated with a spectrum of disease phenotypes ranging from minimal macular involvement with only fleck-like changes to complete chorioretinal atrophy resulting in a generalized cone–rod or rod–cone dystrophy.[8] A common presentation is a precipitous loss of central visual acuity in the first decade of life.[9,10] Patients with early-onset disease (i.e., ≤10 years of age at onset) typically have more aggressive progression,[7] while patients with later onset STGD (i.e., ≥45 years of age at onset) frequently demonstrate foveal sparing[6] with milder progression. The spectrum of severity of STGD has been explained by a wide variety of biochemical defects.[11] Maugeri et al.[5] described a genotype–phenotype model and proposed a classification of combined ABCA4 mutations for each patient as “mild,” “moderate,” and “severe” based on the effect of the mutation on the function of the protein.

There is currently no effective treatment for STGD, but several phase I/II clinical trials are underway. These include a gene therapy trial investigating an Equine Infectious Anemia Virus (EIAV) based lentivector administered by subretinal injection, SAR422459 (NCT01367444) as well as a trial investigating an orally administered vitamin A analog, C20-D3-vitamin A (NCT02402660).[12] With these potential therapeutics on the horizon, it is essential to determine the test–retest variability for the ophthalmic tests commonly used to evaluate disease progression in the severely affected patients likely to enter these early phase trials. Test–retest variability is typically assessed by measuring test–retest repeatability, which has been reported in many ophthalmic pathologies such as glaucoma,[13,14] X-linked juvenile retinoschisis (XLRS)[15] and retinitis pigmentosa (RP).[16-18] To our knowledge, there is currently limited information regarding the test–retest variability in STGD patients in the late stages of their disease.[19,20] These previous studies have examined repeatability of microperimetry and multifocal ERG, but there is currently no published test–retest information regarding visual acuity, visual fields, and structural measures acquired by spectral-domain optical coherence tomography (SD-OCT) for STGD patients.

Typically participants with advanced forms of a disease are chosen as initial participants for Phase I/II clinical trials to demonstrate the safety of an investigational therapy in the early stages of clinical development. The purpose of the study was to measure test–retest variability of several commonly used functional and structural measures in participants with advanced stages of STGD disease who will be likely candidates for phase I/II clinical trials.

Materials and Methods

The data from both eyes for screening and baseline visits prior to treatment in the Phase I/IIa dose escalation safety study of SAR422459 (NCTO1367444) was collected and analyzed. The patients were recruited over a period of 3 years from two sites (Oregon Health & Sciences University, Casey Eye Institute and Centre Hospitalier National d'Ophtalmologie des Quinze-Vingts).The trial conformed to the Declaration of Helsinki for research involving human participants and was approved by the Oregon Health & Science University (OHSU) institutional review boards and the Comité de Protection des Personnes Paris Ile de France V. Informed written consent was obtained from all the participants in the study prior to the conduct of any study procedures.

Participants

The key inclusion criteria for all participants were: (1) a diagnosis of advanced (moderate to severe) forms STGD based on the criteria of Lois et al.[21] and Fishman et al.,[9] (2) two pathogenic mutations of ABCA4 with confirmed parental segregation, and (3) visual acuity less than or equal to 35 to 50 Early Treatment Diabetic Retinopathy Study (ETDRS) letter score in the worse eye.

Key exclusion criteria were (1) media haze, (2) aphakia or prior vitrectomy, (3) other diseases affecting the visual function (e.g., glaucoma, optic neuropathy, active uveitis, retinopathy, and maculopathy other than Stargardt), (4) myopia greater than 8 diopters (D) spherical equivalent, (5) history of ocular surgery within 6 months, (6) concomitant systemic disease in which the disease itself, or the treatment for the disease, could alter ocular function. During the screening and baseline study visits, all participants had a full ophthalmic examination including biomicroscopy, funduscopy, fundus autofluorescence (FAF), full-field ERG as well as the tests analyzed in this publication. All patients had central choriocapillaris and RPE atrophy on the fundoscopic examination (Fishman stages III and IV).[22] With FAF, six patients had type A phenotype with central hypofluorescence surrounded by a hyperfluorescent ring.[4] A majority of patients (16) demonstrated type B phenotype with only central hypofluorescence without surrounding hyperfluorescence[4] (Fig. 1).

Figure 1

Representative images that demonstrate FAF (A, C) and the fundus appearance (B, D) from 42-year-old patient (A and B) and 40-year-old patient (C and D) with advanced STGD both associated with the homozygote mutation on ABCA4 (missense). Image A demonstrates FAF phenotype type A with central hypofluorescence surrounded by a ring of hyperfluorescence. Image C demonstrates FAF phenotype type B with only central hypofluorescence without surrounding hyperfluorescence.

Procedures

BCVA, static, and kinetic perimetry were measured in triplicate (sessions 1, 2, and 3) and SD-OCT was performed twice at the screening and baseline visit (sessions 1 and 2). The visits were a maximum of 3 weeks apart and a minimum of 1 week apart. Data with no numerical value (e.g., zero letters read for BCVA) on two of three tests were excluded from the analysis to lessen the possibility of the underestimation of test–retest variability due to a floor effect.

Visual Acuity

BCVA was measured using the ETDRS chart on the electronic visual acuity tester (EVA) at a distance of 3 meters and recorded as the number of letters read.[23] The right eye was tested prior to the left eye. For subjects with poor central vision, it was suggested that the subject fixate eccentrically or turn or move his/her head in any manner that improved visual acuity. If the subject employed these maneuvers, the technician ensured that the fellow eye remained covered. Subjects were also instructed not to lean forward.

Static and Kinetic Perimetry

Both static and kinetic perimetry were performed using the Octopus 900 Pro (Haag-Streit International, Koeniz, Switzerland) with Eye Suite i4.000 software (Haag-Streit International, Koeniz, Switzerland). The background illumination of the cupola of the Octopus 900 during the testing was 10 cd/m2 (31.4 apostilbs).

Data from static perimetry testing was collected using a custom designed grid (STGD-RP 184n) with stimulus size V, the GATE strategy and a centrally condensed grid of 184 points extending from 56° nasally to 80° temporally. The grid used in this study was of radial design with the closest interstimulus distance centrally and interstimulus intervals increasing with eccentricity toward the periphery. In general, the interstimulus distance increased progressively with increasing eccentricity with the points within the central 30° placed for as even coverage as feasible to accommodate the large central scotomas seen with moderately advanced Stargardt disease. Of the 184 test locations in this grid, 108 occur within 30° radius from fixation. The interstimulus interval within the central 5° radius of the grid is 2.82843° between points at the obliques of 45° to 225° and 135° to 315°. The interstimulus interval at the 10° and 20° radius was 5.2° and 5.8°, respectively. At the 30° radius, the interval varied from 6.2° to approximately 10.3°. Beyond the 30° radius, the grid becomes notably looser with interstimulus intervals of about 17.4° at 55.5° radius from fixation (superiorly and nasally) and approximately 25° at 80° from fixation (temporally). However, as can be seen in the illustration of the grid, staggering of the successive rings of stimulus points effectively reduces the overall stimulus interval to its nearest neighbor. Thus, the test grid was designed with condensation of test locations within the central field for better definition and resolution of central scotomas of relatively small to large size. This graded condensation was extended to 30° to enable definition of large central scotomas (Fig. 4D). The data from this testing was then used to measure retinal sensitivity as a hill of vision (HOV) at 90° as a volumetric estimate of retinal sensitivity, by using visual field modeling and analysis (VFMA) software reported by Weleber et al.[24]

Figure 4

Representative images showing intervisit variability of static perimetry from 25-year-old patient with advanced Stargardt disease associated with the homozygote mutation on ABCA4 (deletion) by using the volume of the HOV model between session 1 (A), 2 (B), and 3 (C). (D) Shows the custom 184n grid pattern for the session 1 on the same patient; (E) generated incremental color coded plot for the session 1 on the same patient.

Data from SKP was collected using stimulus test size V4e, III4e, and I4e and the total seeing area was calculated for each isopter (seeing area minus defined scotoma). Test vectors were presented to the participants approximately every 15°, at an angular velocity of 4°/second, and originating approximately 10° outside the age-correlated normal isopter. Any scotomas (nonseeing areas within a seeing area) were mapped using an angular velocity of 2°/second, originating from the assumed center and using at least 12 vectors. Mapping of the blind spot was done with the I4e test size target. The technician instructed participants to look straight ahead and constantly monitored patient fixation and eye movement throughout the testing. The program discontinued stimulus presentation in case of eyelid closure or fixation breaks. The technician adjusted the subject's eye and head position as needed throughout testing.

Optical Coherence Tomography

After pharmacological dilation (Tropicamide 1%) macular images were obtained of both eyes with the Heidelberg Spectralis SD-OCT system (Heidelberg Engineering, Heidelberg, Germany). High speed 97 horizontal line volume scans centered on the fovea were performed in duplicate at screening and baseline visits within a 2- to 3-week interval. The horizontal and vertical line scans were acquired at 30° magnification and centered on the fovea. All images were obtained with the Heidelberg Spectralis active eye tracking feature, which allows imaging of the same part of the retina during follow-up exams. The first examination was marked as the reference and the follow-up mode, the special feature of the Heidelberg Spectralis SD-OCT system allowing to scan the same part of the retina, was used for follow-up visits.

OCT scan quality was assessed for the use of image averaging, centering of the scan on the fovea, and that the internal limiting membrane (ILM) and Bruch's membrane (BM) were clearly visible permitting manual segmentation, if needed. ETDRS grids were manually centered on the fovea of the initial volume scans by assessment of the component b-scans. Automatic transfer of the ETDRS grid position by the Spectralis software ensured that identical areas were measured on the follow-up scan. Manual segmentation, based on previously published algorithms, was performed to correct automated segmentation errors for boundaries of the ILM and BM.[25,26] Mean central thickness measurements (central circle of 1-mm diameter) and total macular volume were calculated in the macula by using a circular ETDRS grid.

Statistical Analysis

The R statistical language was used to perform all statistical analyses (http://www.r-project.org). Repeatability coefficients (RC) and 95% confidential intervals (CI) were calculated for each parameter by using a hierarchical mixed-effects model[27] and bootstrapping method. Data from both eyes were included. Nonparametric bootstrap samples were drawn 1000 times for each test. A mixed-effects model was fitted to each bootstrapped sample set to estimate within-patient SD and the corresponding RC values. A mixed model can account for potential correlations among measurements within each eye within a patient. The RC is defined as 2.77× or (1.96 × √2) × within-patient SD (i.e., 1/2 length of 95% CI of the changes between two measurements). Dot plots were used to display the changes between two measurements of the same eye within a patient and the bootstrapping estimates of RC coefficient and their 95% bootstrapping CI.

Results

Demographics of the study subjects are summarized in Table 1. The mean age (±SD) of participants was 41.8 ± 12.6 years, ranging from 24 to 66 years. All the subjects were unrelated with the exception of two subjects who were siblings. See Supplementary Tables S1 for a list of mutations.

Table 1

Subjects' Demographics

Repeatability Coefficient and Confidence Intervals

The repeatability coefficients were: BCVA (8.05 letter score), SKP isopters I4e, III4e, and V4e (3478.85; 2488.02, and 2622.46 deg[2], respectively), full HOV VTOT (14.62 dB-sr), central macular thickness and macular volume (4.27 μm and 0.15 mm3, respectively). The RCs and associated CIs are summarized in Table 2.

Table 2

Repeatability Coefficients

The mean BCVA (±SD) was 25 ± 11 ETDRS letters score (Snellen equivalent 20/320), ranging from 6 to 45 letters. There was no significant difference in right versus left eyes in BCVA, paired t-test, P = 0.79. Figure 2 shows changes in BCVA between three visits relative to the initial visit for all subjects. The data points represent 44 eyes from 22 enrolled participants. The RC (red lines) estimates the criteria of significant change as 8.05 letter score with the 95%CI as 9.52 and 6.63 letters score (dotted blue lines.)

Figure 2

Change in BCVA measured at initial (1) and follow-up visits (2 and 3). The red solid lines show the RC estimated by bootstrapping model (8.05 letter score). The data points represent 44 eyes from 22 participants. The blue dotted lines show the lower and upper 95% CI bound (9.52 and 6.63 letters score).

Visual Fields

The mean SKP area measured for isopters I4e, III4e, and V4e and areas of HOV calculated from static visual fields are summarized in Table 3. Figure 4 shows changes in Full HOV of the static perimetry and SKP between three visits relative to the initial visit for all subjects. Criteria for statistically significant changes were found to be the following: SKP isopters I4e, III4e, and V4e (3478.85; 2488.02 and 2622.46 deg[2], respectively) and full HOV, VTOT (14.62 dB-sr). Table 2 summarizes the RCs and associated CIs. The RC and CIs for isopter I4e were greater than for isopters III4e and V4e (demonstrated on Figs. 3B–D). The RC values were very similar between isopters III4e and V4e. Figure 4 shows representative images of intervisit variability observed in the static perimetry by using volumetric measure, VTOT, (in dB-sr) of the HOV model, which is displayed just beneath and to the left of each model. There was no statistically significant difference between the sessions (P = 0.2 or more). The mean values between the sessions are summarized in Table 3.

Table 3

Visual Field Mean Values between the Sessions

Figure 3

Change in HOV static perimetry VTOT (A), SKP I4e (B), SKP III4e (C), and SKP V4e (D) measured at initial (1) and follow-up visits (2 and 3). The data points represent 44 eyes from 22 participants. The red solid lines show the RC estimated by bootstrapping model and the blue dotted lines show the lower and upper 95% CI bound (18.49 and 11.09 dB-sr).

Variability in CMT and MV Measured by SD-OCT

All enrolled subjects showed the typical STGD outer retinal atrophy with foveal involvement. Only three subjects out of 22 had any of the EZ-band visible within the horizontal and vertical line scan. The follow-up mode was only used for 10 of 22 enrolled subjects, data that were not acquired in the follow-up mode due to patient's nystagmus or lack of cooperation were excluded from the analysis. All of the volume scans for all participants required manual correction. Misidentification and incomplete segmentation of BM and ILM were present for almost all lines within the volume scans and were manually corrected. Figure 5 shows changes in central macular thickness and macular volume measured by SD-OCT between two visits relative to the initial visit. The data points represent 20 eyes from 10 participants. The RC (red lines) estimating significant change for CMT was 4.27 μm and for MV was 0.15 mm3.

Figure 5

Change in CMT (A) and MV (B) measured at initial (1) and follow-up visits (2 and 3). The data points represent 20 eyes from 10 participants. The red solid lines show the RC estimated by the bootstrapping model and the blue dotted lines show the lower and upper 95% CI bound.

Discussion

We have described test–retest variability of common functional and structural measures in patients with advanced forms of STGD related to mutations of ABCA4, in the cohort of patients with advanced STGD enrolled in the SAR422459 (NCT01367444) gene therapy clinical trial. A statistically significant change in BCVA was calculated to be a change of 8 letters. Interestingly, the significant change in BCVA is similar to previously published data by Grover at el.[28] reporting seven letters as a significant change in BCVA in subjects with RP. Jeffrey et al.[15] reported a change between four and eight letters in patients with XLRS using a similar statistical method. For comparison, Arditi at el.[29] concluded that 4.5 letters was a limit for a significant change in visual acuity in trained healthy participants. Vanden Bosch at el.[30] reported a change of 3.5 letters or more on the ETDRS chart is significant in participants with macular disease and age-matched healthy subjects; the difference in test–retest variability between normals and patients in that study were not statistically significant.

To our knowledge, there is no currently published data on the variability of VF sensitivity in patients with STGD. Our study found the criteria for significant change for SKP isopters V4e, III4e, and I4e, were 2622.46, 2488.02, and 3478.85 deg[2] (approximately 24%, 27%, and 60% change of the total field), respectively. The values for the smallest test target I4e had the highest test–retest variability, which might be explained by the relatively low visual acuity of our subjects. Bittner et al.[16] examined variability of Goldmann visual field (GVF) for test targets III4e and V4e in 37 patients with RP and reported the variability percentage change ranged up to 20%,[16] findings that are similar to our data. Grobbel et al.[31] assessed test–retest variability on three separate sessions of full-field kinetic perimetry with Octopus 900 on 14 healthy subjects. The test–retest variability was less than 5°. If one assumes a circular visual field with an average radius as 70° for test targets III4e and V4e, then the area of the test–retest variability can be roughly estimated as less than 2200 deg[2] (approximately a 14% change) in healthy subjects.

We report 14.62 dB-sr as a significant change (approximately a 24% change) for VTOT. Weleber et al.[24] found a RC of 6.29 dB-sr (20.9% change) in a cohort of 10 patients (mean age 42.1 ± 16.5 years) with RP of varying inheritance types, stages of disease, and severity. Weleber et al.[23] also reported a RC of 9.81 dB-sr (9.5% change) for a group of 10 healthy controls (mean age 42.1 ± 11.7 years).

The above noted variability in functional measures could arise from several causes. With repeated testing there could be a learning effect. We did note a slight, nonstatistically significant improvement in BCVA and visual field sensitivity from sessions 1 to 3. The variability of functional measures in patients with STGD was higher compared with reported data in healthy participants by using similar test strategies. This difference in variability could be explained by the presence of unstable fixation or intratest changes in the preferred retinal locus (PRL) in STDG population. Lack of registration of retinal visual field sensitivity to the retina is one of the limitations of these functional measures. Continuous monitoring of eye position by the technician helps to minimize the effects of eccentric or variable fixation. However, unlike microperimetry, standard VF testing does not produce a data set that is registered to the retina. Thus, eccentric fixation or use of variable PRLs, each of which is common in patients with STGD disease, will increase variability. The central macular thickness (RC 4.27 μm) and macular volume measures of 0.15 mm3 were very close between the sessions consistent with the lack of progression of retinal atrophy over the short time between the sessions. The technician's role and patient's cooperation are very important to achieve this degree of reproducibility. In this study, only 10 of 22 participants had OCT acquired in the follow-up mode so this high reliability is biased by our selection criteria. Thus, this test might be a challenging end-point parameter for patients with advanced STGD as many patient's fixation will not permit data acquired in follow-up mode. Also, all OCT scans contained segmentation errors such as incomplete segmentation and misidentifications of retinal boundaries. The required manual correction was a very time-consuming procedure. Similar challenges were observed in other previously published papers.[26] In less advanced patients, it might be possible to measure thickness and area of specific subsets of the retinal layers, but this could not be reliably performed in this group of patients with advanced disease. The test–retest variability in central macular thickness and macular volume using SD-OCT of healthy subjects has been reported as 6.5 μm and 0.06 mm3, respectively.[32] Compared with the functional changes the variability of structural changes were less variable and similar to reported data reported in healthy individuals.

One of the limitations of our study is the lack of fixation data. As noted above, fixation is only coarsely controlled with static perimetry as measured in this trial. Cideciyan et al.[20] using the NIDEK MP1 in participants with STGD reported a RC for the point-wise sensitivity (PWS) as 4.2 dB. This group has also reported different fixation types and levels of eccentricity in STGD patients. It would be interesting in the future to compare the MP1 sensitivity in Stargardt patients with their ability to perform static perimetry, as well as to compare and correlate the fixation type and eccentricity with the intervisit variability in visual fields.

This study involved only 22 participants all with advanced Stargardt disease; thus, one must use caution when applying our findings to other STGD populations.

In summary, this test–retest analysis provides important information necessary to determine if significant changes are occurring in structural and functional assessments commonly used to monitor and evaluate outcomes in patients with advanced STGD, the patients most likely to be enrolled in phase I/II interventional studies. These data can be useful for designing future clinical trials and show the challenges and importance of performing repeat testing prior to treatment. Similar analyses should be performed for patients with different stages of disease.

Click here for additional data file.