Interobserver and Intraobserver Agreements of the Detection of Demodex Infestation by in Vivo Confocal Microscopy.

Objectives: The purpose of the study was to determine interobserver and intraobserver agreement, repeatability, and intrasubject variation of the detection of Demodex infestation in eyelids of blepharitis patients using in vivo confocal microscopy (IVCM).
Methods: Eighty-three eyes of 42 blepharitis patients were included in the study. All eyelids were evaluated from temporal to nasal with IVCM using section mode and 10 lashes with their follicles were imagined, and every image with suspicion of Demodex infestation was recorded. Two experienced and two inexperienced ophthalmologists were masked for the diagnosis and interpreted the IVCM images regarding the presence of Demodex infestation with a 3-week interval. Interobserver and intraobserver agreements were calculated with Cohen's kappa and its variant statistics between and within experienced observers and between inexperienced observers.
Results: While average sensitivity for the diagnosis of demodicosis in IVCM images was 83.35% for experienced and 51.35% for inexperienced observers, the average positive predictive value was 88.6% for experienced observers and 91.05% for inexperienced ones. Interobserver agreement between experienced observers was moderate (κ = 0.529) and intraobserver agreements within experienced observers were perfect (κ = 0.918 for observer-1; κ = 0.958 for observer-2). Interobserver agreement between inexperienced observers was poor (κ = 0.162) and intraobserver agreements within inexperienced observers were fair (κ = 0.427 for observer-3; κ = 0.475 for observer-4).
Conclusion: The sensitivity, interobserver and intraobserver agreements in IVCM image analysis for the detection of Demodex infestation were highly associated with the clinical experience on IVCM imaging. In the hands of an experienced clinician, IVCM could be reliable for the diagnosis of ocular demodicosis. Copyright:

Introduction

Blepharitis is the chronic eyelid margin inflammation and one of the most common conditions encountered in ophthalmology clinics. Blepharitis could affect any age and sex group and may result in dry eye disease by damaging the eyelids and the cornea (1). Confusion on the differential diagnosis of blepharitis and its frequent association with dry eye disease complicates to achieve clear information from prevalence studies. A survey of clinicians in the United States showed that 37–47% of adults patients had findings of blepharitis (2). Venturino et al. studied ocular discomfort in 1148 patients and found that 12% of the subjects had anterior blepharitis, 24% had posterior blepharitis, and 21% had dry eye (3). Chronic blepharitis frequently coexists with meibomian gland dysfunction and irreversibly damage to the ocular surface. Mathers et al. observed that 74% of chronic blepharitis patients have meibomian gland dysfunction (4).

The pathophysiology of blepharitis is still unclear, but its association with staphylococcal superinfection is well established (5). In addition to chronic bacterial infection, (6) environmental factors, (7) and systemic diseases, (1) Demodex infestation is also involved in the pathogenesis of blepharitis (8). Recently, a meta-analysis based on 13 published case–control series on the association between Demodex infestation and blepharitis indicated that Demodex mites were related to chronic blepharitis (9).

Demodex, a genus of small parasitic mites affecting mammals, was first identified in 1841 (10). Although it is the most common ectoparasite in humans, it has recently attracted the attention of clinicians, including ophthalmologists, dermatologists, and other specialists (11). Demodex folliculorum and Demodex brevis are the only two species that affect the human eye (12). D. folliculorum has larger body, about 0.3–0.4 mm long, and clusters as groups in the hair follicles (13). D. brevis has a much smaller size, about 0.2–0.3 mm long, lives solitarily in the sebaceous gland (8). Consequently, while D. folliculorum is associated with anterior blepharitis, D. brevis is associated with posterior blepharitis, meibomian gland dysfunction, recurrent chalazia, and refractory keratoconjunctivitis (14). Demodex mites occlude the sebaceous gland orifices, which cause irritation, hyperplasia, and hyperkeratinization in the eyelid margins (15). The cytoskeleton of the mites also may act as a foreign body and cause an inflammatory immune response or granulomatous reaction as implicated in chalazia (16). Furthermore, the secretions of mites may act as a vector that carries bacteria such as staphylococci and streptococci (17). In recent studies, interdependency between demodicosis, staphylococci superinfection, and microbial blepharitis has shown in different patient groups (18). Hereby, accurate, reliable, and reproducible demonstration of Demodex infestation will be useful for diagnosis and follow-up of chronic blepharitis cases.

The diagnosis of demodicosis is mainly based on clinical evaluation and cylindrical dandruff (CD) has been reported to be a reliable diagnostic marker in microscopic examination (19). However, it is difficult to diagnose Demodex infestation, since there is no macroscopic visible finding and light microscopy examination requires eyelash sampling from the patients. Recently, a non-invasive and painless diagnostic technique, laser scanning in vivo confocal microscopy (IVCM) has been used for diagnosis of Demodex infestation and found reliable and effective for ophthalmologic examination (20,21). IVCM is a non-invasive imaging modality that allows real-time imaging of corneal, conjunctival, and meibomian glands structure at a cellular level (22-25). IVCM is also commonly used by dermatologists to diagnose Demodex skin infection (26). Despite its widespread and frequent usage, subjective interpretation of IVCM images could negatively affect its reliability and reproducibility (27,28). The role of the observer in IVCM image analysis has been discussed previously in various disease groups (22,23,29). IVCM images are generally interpreted by experienced clinicians, however, many ophthalmologists do not have sufficient experience on IVCM image interpretation. It is critical to know the accuracy of the image interpretation by these relatively inexperienced clinicians.

This study aims to evaluate the role of observer experience in the interpretation of IVCM images at the presence of Demodex in patients with seborrheic blepharitis.

Methods

Study Subjects

This study included 83 eyes of 42 consecutive seborrheic blepharitis patients with chronic lid margin inflammation and crusting at the base of the eyelashes. This study was conducted according to the Declaration of Helsinki and it was approved by the local ethical committee. For inclusion in the study, the patients signed informed consent forms. Patients between the ages of 18–55 were included and their age and gender of the patients were recorded as the demographics. All patients had a complete ophthalmological evaluation, including visual acuity testing, slit-lamp biomicroscopic examination, IVCM imaging, and Demodex examination was performed. The patients with major ocular disorder, systemic disease, a history of ocular surgery or trauma, the use of contact lenses, or medications that affect the eye were excluded from the study.

IVCM Imaging

All subjects underwent IVCM (Heidelberg Retina Tomography 3 Rostock Cornea Module, Heidelberg Engineering GmbH, Heidelberg, Germany) with a standard procedure by an experienced ophthalmologist (A.S). A disposable sterile protective cap (Tomo-Cap, Heidelberg Engineering GmbH, Heidelberg, Germany) was attached to the microscope, and Dexpanthenol 5% (Recugel, Bausch and Lomb, NY, US) was applied as a coupling agent between the cap and the lens objective. Before the examination, a local anesthetic (proparacaine hydrochloride 0.5%) drop was applied to both eyes. After the examiner asked the patient to look toward the light source, the center of the Tomo-Cap was touched gently onto the lower eyelid. The lower eyelid of all eyes was evaluated with IVCM (from temporal to nasal) and approximately 10 lashes and their follicles were imagined, and every suspected image was recorded (Fig. 1). All IVCM examinations were performed by an expert examiner (A.S) and images were obtained using the section mode. A large number of photographs were taken individually, from the root to the tip of an eyelash, to evaluate all the angles of the CD and meibomian orifice. The sections that do not have thin hair or dandruff density were selected and the same examiner recorded the three high-quality images for each eye, 249 images total, without descriptive or patient information.

Figure 1

Representative in vivo confocal microscopy images of eyelash follicles of seborrheic blepharitis patients. The scale bar is 100 μm. (a) Demodex, which is buried close to the hair, causing dandruff, weakening the bottom of the hair follicle, (b) Demodex buried close to the lash bottom, (c) Demodex infestation, showing three opisthosomas lined up next to each other, (d) single larva buried, (e) two Demodex body, and (f) cluster infestation embedded in the skin close the lash root.

Optical Microscopy Examination

After IVCM imaging, four eyelashes from the lower lid of each eye were epilated for demodicosis diagnosis. Especially, lashes with a significant amount of CD were targeted for epilation. After the epilation of lashes, one drop of 10% potassium hydroxide (KOH) was added on the lashes and a coverslip was placed carefully on them. The samples in KOH were examined using light microscopy at 10× and 40× by an experienced ophthalmologist (C.A.). The number of Demodex in four lashes in total was recorded for each eye. The median value (five per eye) of Demodex count was assigned as the cutoff value to create subgroups with low (1-5) and high load (six and above) for Demodex infestation.

Image Analysis

Four observers who were masked to the clinical data of patients interpreted the IVCM images regarding the presence of Demodex. The first two observers were cornea specialists with more than 3 years of experience in IVCM image interpretation (i.e., experienced observers: Observer-1 and observer-2), and the other two observers were the cornea specialists without any previous experience with IVCM imaging (i.e., inexperienced observers: Observer-3 and observer-4). All observers have read eight full-text articles, original research, and review articles, which were published on IVCM imaging on Demodex infestation from March 2012 to March 2017, to diagnose the presence of Demodex in IVCM images (18,20,26,30-34). After literature review, all IVCM image set was rated by the observers according to the presence of Demodex. All observers determined the results independently when interpreting the IVCM images. The interobserver agreement was evaluated between and within observer groups with different experience levels. In addition to the interobserver agreement, to evaluate the intraobserver agreement, the randomly blended IVCM images were reinterpreted by all observers after 3 weeks. For intraclass correlation, subject eyes were scored from 0 to 3, with the number of images detected by Demodex from this eye being 0 for no Demodex, 1 for 1–5, 2 for 6–10, and 3 if more than 10.

Statistical Analysis

Statistical analysis was performed using SPSS software (version 26, IBM SPSS Statistics, Chicago, Illinois, USA). The Shapiro–Wilk test was performed to confirm the normal distribution of patient demographics. For each observer, sensitivity, specificity, positive and negative predictive values, and false-positive and false-negative rates for the diagnosis of demodicosis were determined. Diagnosis of demodicosis through light microscopy examination of the lashes was accepted as ground truth for binary classification matrix. Cohen’s kappa variant statistics applied to assess interobserver and intraobserver reliability (35). Siegel and Castellan’s kappa applied for two rater comparisons with bias correction, Fleiss’s kappa applied for more than 2 rater comparison. Kappa values for interobserver and intraobserver reliability were interpreted according to Landis and Koch’s guideline (36). To determine the effect of Demodex load on the diagnosis of observers, logistic regression analysis was performed. Furthermore, intraclass correlation was performed between observers through the number of demodicosis diagnosed images for each eye of the patients. P<0.05 was considered statistically significant.

Results

The mean age of the patients was 26.14±9.59 years, and 21 (50%) patients were women and 21 (50%) were male. The median number of Demodex for each eye was 5 and the interquartile range was between 3 and 7. Among all eyes of the blepharitis patients, 8 (9.6%) eyes did not have demodicosis, 43 (51.8%) eyes had low Demodex load, and 32 (38.6%) eyes had high Demodex load (Table 1).

Table 1

The subject characteristics

Subjects	Values
Women (%)	21 (50.0)
Age (mean±SD, year)	26.14±9.59
Demodex count (median [IQR])	5.00 [3.00–7.00]
Presence of Demodex (%)	OD	OS
None	4 (9.5)	4 (9.8)
Low load (1–5)	20 (47.6)	23 (56.1)
High load (>5)	18 (42.9)	14 (34.1)

SD: Standard deviation; IQR: Interquartile range; OD: Oculus dexter; OS: Oculus sinister.

For the first image analysis, the average sensitivity for detecting Demodex in IVCM images was 83.35% for the experienced observers and 51.35% for the inexperienced observers. The average positive predictive value was 88.6% for experienced observers and 91.05% for inexperienced observers. The sensitivity, specificity, positive and negative predictive values, false-positive and false-negative rates for the first image analysis of each observer are shown in (Table 2).

Table 2

The sensitivity, specificity, positive predictive value, negative predictive value, false-positive rate, and false-negative rate of IVCM imaging for the detection of Demodex infestation for both experienced and inexperienced observers

Measurement	Experienced observers		Inexperienced observers
	Observer-1 (%)	Observer-2 (%)	Observer-3 (%)	Observer-4 (%)
Sensitivity	76.0	90.7	70.7	32.0
Specificity	0.0	0.0	25.0	75.0
Positive predictive value	87.7	89.5	89.8	92.3
Negative predictive value	0.0	0.0	8.3	10.5
False-positive rate	100.0	100.0	75.0	25.0
False-negative rate	24.0	9.3	29.3	68.0

While interobserver agreement between experienced observers was moderate (κ=0.529, p<0.001), interobserver agreement between inexperienced observers was slight (κ=0.162, p=0.004). Average Cohen’s kappa for interobserver agreements between experienced and inexperienced observers was 0.198±0.046 (Table 3). Calculated Fleiss’s kappa for overall agreement for all observers was 0.192 (slight intraobserver agreement, p=0.026)

Table 3

Interobserver agreement for detection of Demodex presence and inter-rater reliability comparison in IVCM images in the experienced and inexperienced observer group

Cohen’s kappa (κ) = 0.529	Observer-2 (experienced)
	Demodex (+)	Demodex (-)
Observer-1 (experienced)
Demodex (+)	107	20
Demodex (κ)	37	80
Cohen’s kappa (κ) = 0.162	Observer-4 (inexperienced)
	Demodex (+)	Demodex (-)
Observer-3 (inexperienced)
Demodex (+)	30	71
Demodex (-)	21	122
Cohen’s kappa (κ)	Inexperienced observers
	Observer-3	Observer-4
Experienced observers
Observer-1	0.219	0.194
Observer-2	0.245	0.137

When the observers repeated IVCM image analysis for intraobserver reliability, they exhibited similar sensitivity, specificity, positive and negative predictive values, false-positive and false-negative rates with the first analysis (Table 4). Intraobserver agreements for experienced observers were in almost perfect agreements (κ=0.918, p<0.001 for observer-1; κ=0.958, p<0.001 for observer-2), compared to moderate agreements of inexperienced observers (κ=0.427, p<0.001 for observer-3; κ=0.475, p<0.001 for observer-4).

Table 4

Intraobserver reliability, sensitivity, specificity, positive predictive value, negative predictive value, false-positive rate, and false-negative rate of repeated IVCM image analysis for the detection of Demodex infestation for both experienced and inexperienced observers

Measurement	Experienced observers		Inexperienced observers
	Observer-1	Observer-2	Observer-3	Observer-4
Intraobserver reliability (κ)	0.918	0.958	0.427	0.475
Sensitivity	72.0	88.0	69.3	26.7
Specificity	0.0	0.0	0.0	50.0
Positive predictive value	87.1	89.2	86.7	83.3
Negative predictive value	0.0	0.0	0.0	6.8
False-positive rate	100.0	100.0	100.0	50.0
False-negative rate	28.0	12.0	30.7	73.3

For intraclass correlation, subject eyes were scored from 0 to 3, which are the count of Demodex detected image from this eye. Intraclass correlation measurement was applied for intraobserver reliability, as well as for both first and repeat IVCM image analyses of observers (Table 5). Similar to previous inter-rater reliability calculations, interobserver agreement within experienced observer group was higher (0.788, p<0.001 for first analysis; 0.730, p<0.001 for repeat analysis) than inexperienced observer group (0.349, p<0.001 for the first analysis; 0.469, p=0.002 for the repeat analysis). Besides their higher interobserver agreement levels, experienced observers achieved higher intraobserver agreement levels (0.974, p<0.001 for observer-1; 0.982, p<0.001 for observer-2) compared to inexperienced observers (0.656, p<0.001 for observer-3; 0.740, p<0.001 for observer-4) in intraclass correlation. The logistic regression analysis demonstrated that Demodex loads of the patients did not affect the sensitivity of IVCM image analysis for Demodex detection (p=0.165).

Table 5

Intraclass correlation levels in number of demodicosis diagnosis images per patient between experienced and inexperienced observers for first and repeated image analysis results

Intraclass correlation levels	Experienced Observers		Inexperienced Observers
	Observer-1	Observer-2	Observer-3	Observer-4
Experienced observers
Observer-1	0.974*	0.788†	0.342†	0.279†
Observer-2	0.730§	0.982*	0.524†	0.286†
Inexperienced observers
Observer-3	0.499§	0.542§	0.656*	0.349†
Observer-4	0.476§	0.463§	0.469§	0.740*

Discussion

In our study, we evaluated the sensitivity, specificity, positive and negative predictive values, and inter-rater reliability and the interclass correlation between observers on the diagnosis of Demodex infestation through IVCM images from seborrheic blepharitis patients. In our patient sample population for seborrheic blepharitis, 38 (90.5%) patients had visible Demodex infestation in eyelash follicles according to the light microscopy examination results. Experienced observers diagnosed these patients with demodicosis with high sensitivity (83.35%). However, for inexperienced observers, the average diagnosis sensitivity was significantly lower (51.35%) for the diagnosis of demodicosis. On the other hand, positive predictive values for all observers were at similar levels regardless of experience. The sensitivity for the diagnosis of Demodex with IVCM increased with clinical experience on IVCM. Our result also supported the results of the study by Wang et al. on the interobserver reliability and validity of IVCM imaging for ocular demodicosis (37). While positive predictive value was not affected by the experience of observer, the sensitivity for Demodex diagnosis increased with the years of experience.

While the interobserver agreement between experienced observers was moderate (κ=0.529), both interobserver agreements within the inexperienced group and between experienced and inexperienced observers were poor. Correlatively, intraobserver agreement within experienced observers was far higher than inexperienced observers. Consistency on the diagnosis of Demodex is correlated with the IVCM experience level of the observer. Finally, we validated interobserver agreement results with intraclass correlation levels of observers and we have found a very similar pattern on intraclass correlation levels. Experienced observers achieved significantly higher intraclass correlation levels compared to inexperienced observers. All these interobserver and intraobserver comparisons of Demodex diagnosis results show, precise and consistent IVCM image analysis for ocular demodicosis may require a long period of clinical experience with IVCM.

One of the limitations of our study is the miscalculation of specificity due to the small number of seborrheic blepharitis patients without Demodex infestation. To pass this limitation, we have also measured the intraclass correlation between observers and we found similar results with inter-rater reliability tests. Another limitation of the study is the non-standardized diagnostic threshold for ocular demodicosis in light microscopy. While Filho et al. (37) suggested the presence of more than 1 Demodex in three eyelashes of each eyelid as positive, Wesolowska offered one Demodex infected eyelash from 10 random eyelashes was enough for diagnosis (38). We have selected the second one for our diagnostic criteria in light microscopy examination. Furthermore, the overdiagnosis of Demodex infestation in the patients by experienced observers leads significant decrease in the specificity of the diagnosis. The main reason for the overdiagnosis could be the increased number of keratinized substances, lipid metabolites, and eyelash scales, which are confused with Demodex parts in IVCM imaging.

Blepharitis is a disease highly associated with Demodex. The previous studies have shown the prevalence of ocular demodicosis in blepharitis patients between 67.2% and 83.7% (39-42). Demodex is not just responsible for blepharitis, it has also significant roles in meibomian gland disorder, (43) chalazia, (44) pterygium, (45) and rosacea (46). All these diseases are frequently seen in the community and require continuous follow-up for treatment. Although the relatively expensive disposable Tomo-Caps constitute a disadvantage in terms of wide use by ophthalmologists, IVCM is a very convenient option for screening, diagnosis, and follow-up of ocular demodicosis and its related diseases, because of its non-invasive, reliable, and reproducible nature (47).

Conclusion

As a result, even in different experience levels of observers, IVCM showed high sensitivity, positive predictive value, and reproducibility for the diagnosis of ocular demodicosis and IVCM could be safely used for the diagnosis of Demodex infestation in ophthalmology clinics.