Reliability and Repeatability of Cone Density Measurements in Patients With Stargardt Disease and RPGR-Associated Retinopathy.

Purpose: To assess reliability and repeatability of cone density measurements by using confocal and (nonconfocal) split-detector adaptive optics scanning light ophthalmoscopy (AOSLO) imaging. It will be determined whether cone density values are significantly different between modalities in Stargardt disease (STGD) and retinitis pigmentosa GTPase regulator (RPGR)-associated retinopathy.
Methods: Twelve patients with STGD (aged 9-52 years) and eight with RPGR-associated retinopathy (aged 11-31 years) were imaged using both confocal and split-detector AOSLO simultaneously. Four graders manually identified cone locations in each image that were used to calculate local densities. Each imaging modality was evaluated independently. The data set consisted of 1584 assessments of 99 STGD images (each image in two modalities and four graders who graded each image twice) and 928 RPGR assessments of 58 images (each image in two modalities and four graders who graded each image twice).
Results: For STGD assessments the reliability for confocal and split-detector AOSLO was 67.9% and 95.9%, respectively, and the repeatability was 71.2% and 97.3%, respectively. The differences in the measured cone density values between modalities were statistically significant for one grader. For RPGR assessments the reliability for confocal and split-detector AOSLO was 22.1% and 88.5%, respectively, and repeatability was 63.2% and 94.5%, respectively. The differences in cone density between modalities were statistically significant for all graders. Conclusions: Split-detector AOSLO greatly improved the reliability and repeatability of cone density measurements in both disorders and will be valuable for natural history studies and clinical trials using AOSLO. However, it appears that these indices may be disease dependent, implying the need for similar investigations in other conditions.

Adaptive optics scanning light ophthalmoscopy (AOSLO) enables noninvasive imaging of the cone photoreceptor mosaic in the living human eye.[1,2] Both confocal and more recently, (nonconfocal) split-detector AOSLO imaging can be undertaken simultaneously[3] and thus, precise spatial registration between the two modalities. Visualization of photoreceptors with confocal imaging is thought to require waveguiding photoreceptors that are relatively intact with correctly oriented outer segments.[3,4] Conversely, split-detector AOSLO resolves cone inner segments regardless of the outer segment integrity.[3] This is especially valuable in diseases where outer segments are not intact, to determine whether there is a lack of cones or whether the cones are present but not waveguiding[3,5] or incorrectly oriented. Quantitative metrics currently used to describe the cone mosaic, such as cone density[6-8] and Voronoi geometry,[9] rely on accurate cone identification. The clinical application of each of these metrics ultimately depends on their reliability and repeatability. Moreover, given the development of gene replacement strategies for patients with Stargardt disease (STGD) and X-linked retinitis pigmentosa GTPase regulator (RPGR)–associated retinopathy, there is an urgent need to reliably assess cone structure in order to ultimately identify sensitive and reliable clinical trial endpoints and improved participant stratification.[10,11]

The reliability of cone density metrics in retinal diseases is inherently limited, as cone identification is more challenging compared to normal mosaics.[7] It is crucial to assess each disease independently, as it is possible that reliability and repeatability will vary across conditions depending on the pattern of degeneration.[7,12,13] Moreover, it is important to assess how reliability and repeatability are influenced by each imaging modality, as it has been shown that absolute estimates of cone density obtained from the two can differ.[8]

Here, we assessed the reliability and repeatability of cone density measurements by using confocal and split-detector AOSLO and determined whether cone density values are significantly different between modalities in patients with STGD and RPGR-associated retinopathy.

Methods

Patients

The study adhered to the tenets of the Declaration of Helsinki and was approved by the Moorfields Eye Hospital Ethics Committee. Informed consent was obtained from all participating subjects after explanation of the nature and possible consequences of the study before enrolment. Images from 12 patients with STGD (8 females and 4 males, aged 9–52 years) and 8 patients with RPGR-associated retinopathy (8 males, aged 11–31 years), were recruited in this study. Patients had varying degrees of disease severity, as a heterogeneous population was chosen. Axial length measurements were obtained with an IOL Master (Carl Zeiss Meditec, Dublin, CA, USA) to calculate the lateral scale of each retinal image.[14] Tables 1 and 2 summarize the STGD and RPGR patient demographics, respectively.

Table 1

STGD Patient Demographics

Table 2

RPGR Patient Demographics

AOSLO Image Acquisition and Processing of the Photoreceptor Mosaic

AOSLO was carried out with a custom-built instrument, previously described.[1] Pupils were dilated before testing with one drop of tropicamide 1% and phenylephrine 2.5% and each patient was stabilized by using a dental impression on a bite bar. Confocal and split-detector image sequences focused on the photoreceptor mosaic were acquired simultaneously. Each image sequence, composed of 150 frames and 20 to 100 videos, was taken for each patient. Various retinal locations up to 7° eccentricity were imaged by using either a 1° or 1.5° field of view. Although the AOSLO operator obtained all the image sequences for each patient, the operator was not the same for all the patients involved in the study.

Following image acquisition, all image sequences were processed by using a previously described strip registration method.[15] Preprocessing steps were undertaken to generate a single registered average image with an increased signal to noise ratio, using a minimum of 50 frames with the highest normalized cross-correlation to selected reference frames from each image sequence consisting of 150 frames. The confocal and split-detector AOSLO images were simultaneously montaged manually with Adobe Photoshop (Adobe Systems, Inc., San Jose, CA, USA) to create a continuous photoreceptor mosaic of the imaged retinal locations.

Retinal areas of 100 × 100 μm were cropped from confocal AOSLO and corresponding split-detector AOSLO montages of acceptable quality for analysis. These images were required to have at least 10 identifiable cones in both modalities and were of varying eccentricities from the fovea in order to include a range of cone densities. Sample AOSLO images used for analysis are shown in Figure 1. In total, the data set consisted of 1584 assessments of 99 STGD images (each image in two imaging modalities, and four graders who graded each image twice) and 928 RPGR assessments of 58 images (each image in two imaging modalities, and four graders who graded each image twice).

Figure 1

Sample 100 × 100-μm AOSLO images used for analysis. (A, B) Confocal AOSLO images of an STGD patient. (C, D) The corresponding split-detector AOSLO images. (E, F) Confocal AOSLO images of an RPGR patient. (G, H) The corresponding split-detector AOSLO images.

Analyzing the Cone Mosaic

Each imaging modality was evaluated independently. Four graders were ranked by experience in AOSLO image analysis such that grader 1 had the most experience and grader 4 had the least. The four graders manually identified cones in each image after standardized training on cone morphology in normal and diseased retina, and also in the use of a customized MATLAB program (Mathworks, Natick, MA, USA). Both dark and bright cones were identified during the confocal analysis, as the dark spaces have been shown to harbor inner segments.[3] The MATLAB program facilitated cone identification by allowing the grader to adjust the image brightness and contrast to assist in determining the presence of a cone. The program kept track of the number and location of the cones selected. The cone-counting interface for manual identification is shown in Figure 2.

Figure 2

Cone-counting interface for manual identification of cones. (A) Example confocal AOSLO image. (B) Example split-detector AOSLO image. Red markers indicate identified cones (top panel). The image brightness and contrast was adjusted by the grader to assist in determining the presence of a cone (bottom panel).

Each grader assessed each image twice for a total of eight trials per image for both modalities. The images were presented in a random order and masked fashion whereby the patient and retinal location were unknown to the grader. Each of the four graders analyzed the randomly presented images at his or her own pace, and breaks were taken as required such that the effect of fatigue was captured by the grader's variance component. The cone counts from each image were then compiled and analyzed. Cone density was determined by dividing the total number of bound Voronoi regions by the total bound Voronoi cell area.[16] Unbounded Voronoi regions were excluded from the analysis.

Statistical Methods

To explore the effects of the grader and subject on cone density measurements from images derived from two modalities, a linear mixed model was fitted with random grader and subject effects. This model splits the variance of cone density measurements into three sources: variance attributed to the (1) grader, (2) subject, and (3) residual variance not attributed to either grader or subject (i.e., measurement error). The better modality is expected to show smaller percentage variance attributed to both the grader and measurement error. Each image was assessed twice by the same grader and the second assessment was consistently higher. The summary of proportions attributed to different variance components is reported in Table 3. The statistical model behind Table 3 calculations accounted for the effect of second assessment to eliminate the discrepancy between the first and second assessments. Since STGD and RPGR cone densities were drastically different, the linear mixed model also controlled for this difference.

Table 3

Percentage of Variance Components

Using the data from Table 3, the reliability was defined as a ratio of variance attributed to image to the total variance, and the repeatability as a ratio of variance attributed to both subject and grader to the total variance.

Bland-Altman analysis and Wilcoxon signed rank tests were used to assess the agreement of cone density values between pairs of confocal and split-detector AOSLO images of the same location. Intraclass correlations (ICCs) were used to quantify intragrader reliability for each of the graders for every disease by modality combinations. In these settings ICC represents percentage variability attributed to the subject. The closer the ICC to 1, the better the reliability.

Results

Table 3 reports contributions of grader, subject, and measurement error to the total variance for STGD and RPGR-associated retinopathy under different modalities.

Table 4 demonstrates that there are differences between the graders and that the modality has an impact on the overall repeatability and reliability. In all cases, the ICC is improved in split-detector AOSLO images and is also less variable across the four graders, compared with confocal AOSLO.

Table 4

Intergrader Reliability as Assessed by the Intraclass Correlation Coefficients

Stargardt Disease

The measured confocal and split-detector cone density values ranged from 960 cones/mm2 to 42,637 cones/mm2, and 755 cones/mm2 to 31,058 cones/mm2, respectively.

Considering only confocal AOSLO, the primary contribution to variability was attributed to the subject (67.9%). The secondary contribution to variability was due to measurement errors (28.8%). The grader had the least contribution (3.3%). In contrast, split-detector AOSLO showed that the grader (1.4%) and measurement errors (2.7%) contributed the least to variability, and the subject (95.9%) contributed the most. The difference for each variance component was statistically significant (P < 0.0001). Accordingly, the reliability for confocal AOSLO and split-detector AOSLO was 67.9% and 95.9%, respectively; and the repeatability was 71.2% (3.3% + 67.9%) and 97.3% (1.4% + 95.9%), respectively.

Figure 3 shows the Bland-Altman plots for all four graders, showing the agreement of cone density values between pairs of STGD confocal and split-detector AOSLO images of the same location. All four graders showed that variability did not change with the magnitude of measurement. The differences in cone density were statistically significant only for grader 1 (P < 0.001) where cone density was underestimated in the confocal images compared with the split-detector images. The differences in cone density were not statistically significant for the other three graders (grader 2: P = 0.206; grader 3: P = 0.235; grader 4: P = 0.512).

Figure 3

Bland-Altman plots for all four graders, showing the agreement of cone density values between pairs of STGD confocal and split-detector AOSLO images of the same location. Solid line represents average mean difference. Dashed lines represent 95% confidence limits of agreement. All four graders show that variability did not change with the magnitude of measurement.

RPGR-Associated Retinopathy

The measured confocal and split-detector cone density values ranged from 3339 cones/mm2 to 38,467 cones/mm2, and 3394 cones/mm2 to 41,977 cones/mm2, respectively.

Considering only confocal AOSLO, the primary contribution to variability was attributed to the grader (41.1%). The secondary contribution to variability was due to measurement errors (36.8%). The subject had the least contribution (22.1%). In contrast, split-detector AOSLO showed that the grader (6.0%) and measurement errors (5.5%) contributed the least to variability, and the subject (88.5%) contributed the most. The difference for each variance component was statistically significant (P < 0.0001). Accordingly, the reliability for confocal AOSLO and split-detector AOSLO was 22.1% and 88.5%, respectively; and the repeatability was 63.2% (41.1% + 22.1%) and 94.5% (88.5% + 6%), respectively.

Figure 4 shows the Bland-Altman plots for all four graders, showing the agreement of cone density values between pairs of RPGR confocal and split-detector AOSLO images of the same location. Graders 1 and 4 showed a negative trend of differences correlated to the magnitude of the measurement such that as cone density increased, the more confocal AOSLO relatively underestimated the cone density measurement, resulting in a larger difference between the two modalities. Graders 2 and 3 showed that variability did not change with the magnitude of measurement. All four graders showed that on average, cone density measurements were underestimated in the confocal images compared with the split-detector images. The differences in cone density were statistically significant for all graders (grader 1: P < 0.001; grader 2: P = 0.003; grader 3: P < 0.001; grader 4: P < 0.001).

Figure 4

Bland-Altman plots for all four graders, showing the agreement of cone density values between pairs of RPGR confocal and split-detector AOSLO images of the same location. Solid line represents average mean difference. Dashed lines represent 95% confidence limits of agreement. Graders 1 and 4 show a negative trend of differences correlated to the magnitude of the measurement. Graders 2 and 3 show that variability did not change with the magnitude of measurement.

Discussion

Quantitative analysis of the photoreceptor mosaic in patients with inherited retinal diseases is challenging, as cones are not as readily identifiable as in healthy eyes. There are multiple contributing factors affecting the reliability and repeatability of cone density measurements,[7,8,17] with reliability being highly dependent on the magnitude of measurement errors.[7] In this study, we estimated the impact of the imaging modality and grader on the repeatability and reliability of cone density measurements by using images from two inherited diseases prioritized for intervention.

An important contributing factor to reliability and repeatability demonstrated in our study was the imaging modality. Graders performed less reliability when using confocal AOSLO versus split-detector AOSLO images, with substantial differences between graders. The grader variance component was larger in RPGR-associated retinopathy than in STGD for confocal AOSLO and less so in split-detector AOSLO. This is likely due to the greater uncertainty when interpreting reflective signals in the confocal images during the cone identification process. The images analyzed had diverse cone densities, as they were of varying eccentricities from the fovea, which may have also impacted the measurement errors. Densely packed photoreceptors in confocal images make it more difficult to distinguish whether a bright spot represents a rod, cone, or other structure, thus impacting the overall reliability—in view of the different patterns of disease progression in STGD compared to RPGR-associated retinopathy, this is more likely to have posed greater challenge in the RPGR images.

Importantly, it is evident that in both RPGR-associated retinopathy and STGD, the grader has less of an impact when using split-detector AOSLO. While there were differences in the repeatability across the four graders, all showed a significant improvement in repeatability when using split-detector AOSLO versus confocal AOSLO. Given that confocal and split-detector AOSLO resolve waveguiding photoreceptors and cone inner segments, respectively,[3,4] the images produced by each modality, although of the same retinal location, are likely to not appear to show an equivalent number of identifiable cones in degenerating retinas. Our data support that split-detector AOSLO is better than confocal AOSLO in capturing the true differences in the data that are attributable to the patient—although in direct contrast to the superiority of split-detector AOSLO for cone density measurements, confocal AOSLO allows assessment of photoreceptor reflectance profiles,[15,18] which is not afforded by split-detector AOSLO. Cone reflectivity may be an important indicator of relative cone structural health in the assessment of photoreceptor integrity.[18] However, in nondiseased eyes, there are cones that appear to be functionally normal yet show low reflectance.[19]

There was substantial variance between graders, which may in part relate to the fact that graders with various levels of experience were used. Similar results were seen when cone density in achromatopsia was assessed.[8] Strong grader effects and a learning effect were displayed, as the second set of cone density values were substantially different from the first. Cones may not be as easily identifiable in a diseased retina with an abnormal photoreceptor mosaic and this may have negatively impacted the repeatability of a less experienced grader. It is therefore crucial that the grader is trained to analyze images of eyes with specific retinal disease, as graders will still be required to review the results of automated methods—in keeping with established disease-specific protocols in reading centers.

Regardless of the imaging modality, the STGD images consistently showed a higher repeatability than the RPGR images, which showed a greater variation among graders, thereby indicating that repeatability is disease dependent. Both STGD and RPGR-associated retinopathy are progressive and display “transition zones” resulting from nonuniform cone loss across the retina. Differences in the underlying pathophysiology, including the extent or pattern of the photoreceptor degeneration—as would be expected in STGD compared to RPGR-associated retinopathy—determine the degree and pattern of disruption of the photoreceptor mosaic integrity, leading to varying levels of inconsistent and ambiguous cone reflectivity in AOSLO images. Common characteristics such as fixation location and stability (e.g., less stable fixation in STGD than RPGR-associated retinopathy)[10] or refractive error (e.g., often high degrees of myopia in RPGR-associated retinopathy compared to STGD)[11] can also impact image acquisition and subsequent analysis.

Image quality also affected cone identification in both confocal and split-detector AOSLO. Poor image quality makes cone identification more challenging. In addition, AOSLO operator differences in image acquisition between patients may also lead to variability yet would not lead to differences between the diseases, as the operator was not the same for all patients imaged in each disease group. In general, given the optical design of the AOSLO, the age of a patient may also play a role in the variability of cone identification, whereby image quality may be degraded by a small pupil diameter, lens opacities, tear film abnormalities, or age-related pathology.[7,20]

Semiautomated algorithms for cone identification have shown that reliable cone density measurements can be obtained between graders and between instruments in healthy eyes, yet the development of modified automated algorithms is crucial to account for factors that affect repeatability and reliability in eyes with abnormal photoreceptor mosaics.[7,21]

In conclusion, we showed that split-detector AOSLO significantly improves the reliability and repeatability of cone density measurements and will be valuable for natural history studies and clinical trials using AOSLO. Understanding the grader differences will require further investigation to identify the underlying contributing factors. Refining and establishing detailed training and standardized protocols will lead to increased measurement reliability in order to take the first step toward a reading center–based format of AOSLO analysis for future large multicenter trials. This is of particular importance when assessing disease progression at the cellular level or when determining treatment areas and the efficacy of potential therapeutic intervention.