Ability of a neuro-ophthalmologist to estimate retinal nerve fiber layer thickness.

BACKGROUND: Qualitative description of the optic disc has clinical value, but optical coherence tomography (OCT) has provided the ability to quantify retinal nerve fiber layer (RNFL) thickness.
METHODS: We asked three neuro-ophthalmologists of at least 20 years' experience to estimate the average OCT RNFL thickness of 37 eyes based on fundus photos.
RESULTS: The overall correlation coefficient for RNFL thickness estimation variance between two physicians and between physician and OCT was 0.53. The likelihood that the RNFL thickness estimation between physicians, or between physician and OCT, was within 10 μm of each other was 47%-62%. All physicians had disparities in RNFL thickness estimation greater than 30 μm.
CONCLUSION: This study provides information on the ability of an experienced neuro-ophthalmologist to estimate the RNFL thickness based on fundus photos.

Introduction

Optical coherence tomography (OCT) is an efficient method of quantifying retinal nerve fiber layer (RNFL) thickness. RNFL thickness can be used to measure atrophy after optic neuritis1,2 and as a marker for generalized central nervous system degeneration in multiple sclerosis.3–5 RNFL thickness in eyes with optic atrophy reflects both the remaining functional nerves and residua such as glial remnants or nonfunctioning axons.6 These anatomic measurements have a physiologic correlation, as average RNFL thickness decreases in proportion to visual field loss after optic neuritis, probably reflecting loss of functional axons.2 Thus, the objective measurements found with OCT are becoming more important in the setting of performing clinical trials. The range of RNFL thickness in nonedematous eyes is roughly 110 μm at maximum, and at minimum, as in eyes with no light perception after longstanding optic neuropathy, average RNFL thickness is approximately 45 μm.7

Our objective was to compare the ability of experienced neuro-ophthalmologists to estimate average RNFL thickness of eyes based on fundus photos showing the optic discs and peripapillary region and compare their result to the OCT measurement. We anticipated that agreement for physician estimates of RNFL thickness from fundus photos would not be superior to concordance measurements previously found for OCT, but we were particularly interested in the extent of correlation, as there has been no previous study quantifying this measurement.

Methods

Institutional review board approval for this study was granted by a university internal review board. Data present in this manuscript were, in part, presented as an abstract for the 2012 North American Neuro-Ophthalmology Society Meeting in San Antonio, TX, as Poster 55.

Digital photographs and OCT were obtained from 37 eyes, which were selected based on gathering a range of RNFL thickness between 35 and 110 μm. We included eyes in which fundus photos and OCT were both performed on the same day. Fundus photos including optic disc and peripapillary region were all performed with a Visucam digital camera (Carl Zeiss, Thornwood, NY) using a standard color capture mode with a −2 correction flash and 30° angle. OCT was acquired using a Carl Zeiss Stratus OCT Fast RNFL thickness profile to generate an average RNFL thickness. Only OCT RNFL measurements with a manufacturer’s signal strength score of at least 7 with a properly centered scan circle were selected.

Photos of the optic discs were digitized and sent through electronic mail to three practising neuro-ophthalmologists with clinical experience of 21, 26, and 29 years, respectively, who were asked to estimate the OCT average RNFL thickness based only on the standard color fundus photos. The three estimating physicians were not the authors of this study, nor had they previously seen the fundus photos or subjects whose eyes were used. No clinical or paraclinical data were available for the physician to produce an estimation. Prior to data collection it was determined that an estimation of under 75 μm would be used as a threshold of disc atrophy. If an OCT estimation was recorded as under 75 μm, it would be labeled as “atrophic.” An OCT estimation over 75 μm would be labeled as “not atrophic.” This thickness threshold was based on a value of less than or equal to 75 μm, being the OCT thickness corresponding to approximately the fifth-lowest percent of normal.

Statistical analysis

We used the Bland–Altman method to calculate the limits of agreement between the RNFL thickness estimation of each physician and the OCT RNFL thickness. To obtain a specific index to summarize the agreement degree, we used both Shrout and Fleiss intraclass correlation coefficient (ICC) and Lint’s concordance correlation coefficient (CCC).8 Barnhart’s overall CCC was used to calculate agreement of RNFL thickness estimation among the three physicians. Shrout and Fleiss ICC was used to calculate the ICC.9 A Levene test was used to check the homogeneity assumption for the three ICCs.

Results

Thirty-seven fundus photos (16 right eyes and 21 left eyes) from 26 subjects (17 female, nine male) were studied. Both eyes were used from nine subjects, and for the remaining 19, one eye was used. The median patient age was 49 years (range 20–75 years). Table 1 reports the number of eyes for each OCT-determined RNFL thickness range as well as the primary diagnosis for each eye.

Table 1

Eyes used for photo estimation and OCT analysis, including the number of eyes used within each RNFL thickness range, and the primary diagnosis for each eye

OCT RNFL thickness range (μm)	Number of eyes	Primary diagnosis
30–40	1	MS and drusen
40–50	1	ON of unknown etiology
50–60	10	3 drusen
		5 nonarteritic ischemic ON
		2 MS
60–70	11	3 drusen
		2 NA-AION
		1 MS
		1 optic neuritis and MS
		3 ON of unknown etiology
70–80	5	2 NA-AION
		1 tobacco-alcohol amblyopia
		2 diabetic papillitis
80–90	3	1 fellow eye with NA-AION
		2 headache
90–100	5	3 MS
		2 headache
100–110	1	MS

Abbreviations: MS, multiple sclerosis; NA-AION, nonarteritic ischemic optic neuropathy; OCT, optical coherence tomography; ON, optic neuropathy; RNFL thickness, retinal nerve fiber layer thickness.

The agreement index compared with OCT based on the CCC for physician A was 0.53 (95% confidence interval [CI]: 0.30–0.70). For physician B, agreement index was 0.43 (95% CI: 0.14–0.65). For physician C, agreement index was 0.72 (95% CI: 0.53–0.84). CCC and ICC data are represented in Table 2. Figure 1 displays CCCs for the three physicians. The overall CCC reflecting agreement among the three physicians was 0.53 with a 95% bootstrap CI of 0.29–0.69. Two thousand bootstrap samples created from the original data set with replacement were used to create the 95% CI.

Table 2

CCC and ICC values with 95% confidence intervals for each physician compared with OCT, and overall coefficient, which combines all three physicians

	CCC	CCC 95% CI	ICC	ICC 95% CI
Dr A and OCT	0.53	0.30–0.70	0.54	0.27–0.73
Dr B and OCT	0.43	0.14–0.65	0.43	0.14–0.66
Dr C and OCT	0.72	0.53–0.84	0.72	0.51–0.85
Overall coefficient	0.53	0.29–0.69	0.53	0.34–0.70

Notes: For both CCC and ICC, the closer a value to 1.00, the higher the degree of correlation.

Abbreviations: CCC, concordance correlation coefficient; CI, confidence interval; ICC, intraclass correlation coefficient; OCT, optical coherence tomography.

Figure 1

(A) Correlation graphs of retinal nerve fiber layer (RNFL) thickness estimations for physicians A, B, and C compared with optical coherence tomography (OCT). An exact reproducibility between physician and OCT results in a point plotted on the 45° graph line. A physician overestimation compared with OCT plots above the line, and an underestimation plots under the line. The further away a point is from the 45° line, the greater the difference between the RNFL estimation of the physician from the OCT. Below, percent of eyes (x-axis) that had RNFL thickness estimation agreement for each range in the y-axis between (B) two physicians and (C) physician and OCT.

OCT RNFL thickness ranged from 37.7 μm to 104.6 μm. Bland–Altman analysis determined a limit of agreement for OCT and physician A of −32.57 to 42.96. Limit of agreement for OCT and physician B was −33.09 to 40.77. Limit of agreement for OCT and physician C was −27.70 to 18.89. Figure 2 displays Bland–Altman plots for the three physicians.

Figure 2

Bland–Altman graph representing limits of agreement for physicians A, B, and C compared with optical coherence tomography (OCT).

Notes: Average OCT for each comparison is plotted on the x-axis; difference between the physician and OCT score is on the y-axis, calculated as physician – OCT. Twice above and below standard deviation of the means is represented by the solid lines. The further away from zero a point plots on the y-axis, the greater the difference that point is between the rater and OCT.

Abbreviation: RNFC, retinal nerve fiber layer thickness composite.

Between physicians, the likelihood was 57.7% for estimating an RNFL value within 10 μm of each other, and 83.8% that the estimate would be within 20 μm of each other. There was a 52.3% likelihood among all physicians for an estimation within 10 μm of the OCT, and 82.0% for an estimation within 20 μm between physician and OCT. Table 3 displays the range of variation among physicians and OCT. The chance of underestimating and overestimating the OCT result for all physicians was equal at 50%.

Table 3

Range of RNFL thickness estimation difference between two physicians and between each individual physician and OCT

	Percent of estimations differing by 0–10 μm	11–20 μm	21–30 μm	>30 μm
Dr A and Dr B	62%	22%	8%	8%
Dr B and Dr C	51%	35%	3%	11%
Dr A and Dr C	59%	22%	8%	11%
Dr A and OCT	47%	31%	8%	14%
Dr B and OCT	51%	27%	8%	14%
Dr C and OCT	59%	32%	5%	3%

Abbreviations: OCT, optical coherence tomography; RNFL thickness, retinal nerve fiber layer thickness.

Twenty-eight photos represented eyes with an OCT RNFL thickness of 75 μm or less. Sixty-three of 84 (75%) of the estimations based on these photos were rated as 75 μm or less by the physicians. Forty-one of 51 (80%) of the estimations based on photos of eyes with an OCT RNFL thickness of less than 65 μm were rated as 75 μm or less by the physicians. An OCT RNFL thickness measurement of 53 μm was the threshold of thinning after which all physicians estimated the RNFL thickness as 75 μm or less. Fifteen of 15 (100%) of physicians rated these RNFL thicknesses as less than or equal to 75 μm. Eyes with OCT RNFL thickness measurements of greater than 53 μm had at least one physician RNFL thickness estimation of 75 μm or more.

Nine photos represented eyes with an OCT RNFL thickness of greater than 75 μm. Nineteen of 27 (70%) estimations based on these photos were rated as greater than 75 μm by the physicians.

Discussion

To our knowledge, there has been no prior study examining the ability of a neuro-ophthalmologist to estimate the average peripapillary RNFL thickness found on OCT, based on fundus photos. Nor has there been a study to compare this estimation with that of other neuro-ophthalmologists of similar clinical experience.

In a different study from ours, physicians detected optic nerve pallor with a sensitivity of 32% and specificity of 92% for eyes in which pallor was defined as having an average OCT RNFL thickness under the fifth percentile of normal. For eyes with an average RNFL thickness under the first percentile of normal, sensitivity improved slightly to 44%, and specificity was 91%. Estimators in this study ranged in experience from residents to attending neurology faculty, and direct ophthalmoscopes were used.10 Although that study serves as a real-world estimation for the likelihood of detecting optic atrophy, ours is a more specific examination of the ability to estimate the OCT RNFL thickness.

We did not ask our estimators to determine whether a disc was “atrophic” or not. Nor did we report to them the RNFL thickness that we would consider “atrophic.” By defining optic atrophy in our study as an OCT value of less than or equal to 75 μm, a thickness corresponding to approximately the fifth-lowest percent of normal, our estimators had a sensitivity of 75% in estimating RNFL thickness under 75 μm when the OCT RNFL thickness was less than or equal to 75 μm. When the OCT RNFL thickness was 65 μm or less, the sensitivity increased to 80% that the physician would estimate the RNFL thickness as 75 μm or less. The RNFL thickness in which there was 100% sensitivity that our physicians would estimate the RNFL thickness at 75 μm or less was 54 μm on OCT.

Some clinical scales, such as the Kurtzke functional system scores, and the expanded disability scale for multiple sclerosis, use the presence or absence of optic atrophy as a factor of rating disability. Using the cut off that an OCT thickness of 75% normal represents atrophy, our results suggest that even in experienced raters, up to 25% of discs may be discordantly interpreted between two physicians. Complementing clinical scales with OCT may serve to minimize variability.

CCC and ICC measure reproducibility, where the degree of reproducibility improves as the agreement index approaches 1.0. Prior studies have evaluated the reproducibility of OCT between trials and between examiners. Cettomai et al11 reported an ICC of 0.89 for interrater reproducibility of OCT measurements with a 95 CI for each eye ranging between 0.72 and 0.96. This ICC result for OCT outperforms the overall reproducibility ICC of our three physicians of 0.56 (95% CI: 0.38–0.72). CCC has some advantages over ICC for correlation analysis; thus, we also calculated CCC scores, which did not differ significantly from ICC. It should be noted that when variance of the sample exceeds that of the population, both CCC and ICC may overestimate agreement. We did not expect physician coefficients to be superior to OCT, but results showed that physician estimations were often very close to OCT, although at times were widely discordant.

Despite many advantages, there remain several shortcomings to OCT for evaluation of the optic disc. Tilted discs or healthy, normal-appearing discs with anomalous peripapillary choroidal degeneration may show thinning of the RNFL on stratus OCT.12 Spectral-domain OCT may be superior in this setting. Nevertheless, without a corresponding clinical exam, over-reliance on OCT results could lead to erroneous extrapolations of optic nerve health. Also, disc edema is presently suboptimally measured by OCT due to limits of recording, and the Frisen scale remains the standard for grading papilledema. Several studies have compared Frisen grade with OCT-measured edema, with mixed results.13 To avoid this issue, we did not use edematous discs in our investigation.

Limitations of our study include the use of fundus photos as a surrogate for real-life viewing of the fundus to estimate the OCT average RNFL thickness. Fundus photos may be inferior to binocular viewing of the disc, but practicality in study design mandated the use of photos. Second, we sampled only three neuro-ophthalmologists as estimators. Clearly, more estimators would have provided a greater sampling of data, but by using a relatively large number of eye estimations per physician, we were able to obtain significantly meaningful data. Third, we estimated average OCT RNFL thickness which rarely represents evenly distributed atrophy. Usually there is sectoral (focal) atrophy with other areas of relatively normal RNFL thickness. We did not ask for estimations of sectoral thickness in our study, but this is a possible point for future investigation.

Conclusion

Although neuro-ophthalmologists are quite capable of estimating average OCT RNFL thickness based solely on fundus photos, responses can vary widely, and up to 20% of the time, the responses will disagree with each other or with OCT by at least 20 μm. We underscore that OCT is a valuable objective tool that is meant to supplement, not replace, the clinical exam.