Self versus examiner administration of the Ocular Surface Disease Index©.

PURPOSE: To compare the difference in Ocular Surface Disease Index© (OSDI) scores when participants were given the OSDI to complete on their own (self-guided, SG), versus under the guidance of the examiner (examiner-guided, EG).
METHODS: 100 participants enrolled in this prospective two-visit study (fifty under-45 years old, 38F/12M; and fifty 45 years-and-older, 42F/8M). Participants who scored ≥1 on the Subjective Evaluation of Symptoms of Dryness (SESoD) were included in this study. Participants completed the OSDI SG during the first visit. Participants returned the next day and repeated the OSDI, but with EG (with standardized instructions). Participants were under deception and believed that they were comparing the OSDI to the SESoD.
RESULTS: The mean OSDI score of the SG and EG administration was 32.0±17.3 and 33.8±19.6 respectively (p>0.05) with 95% limits of agreement between -20.6 and +24.2. The correlation between SG and EG administration was Spearman's r=0.81, p<0.01. The mean difference between SG and EG was not significant (p>0.05) for both the under-45 group, and 45-and-older group. The 95% limits of agreement for the under-45 group were smaller than the 45-and-older group (under-45: [-15.5, +13.1,], 45-and-older: [-23.3, +32.2]). A significant difference was found between 8 of the 12 questions items (all p≤0.01). However, the mean difference for each was <0.6 and was not considered to be clinically significant.
CONCLUSION: There was no clinically significant difference in OSDI score between SG and EG administration, however having instructions provided with EG administration affected variability of scores in the older group more than the younger group.

Introduction

It is well recognized that dry eye syndrome (DES) is a condition driven mainly by symptoms, and that DES can have a negative impact on quality of life. While the symptoms of DES have been repeatedly shown to have poor correlation with diagnostic signs,2, 3, 4, 5 dry eye questionnaires are frequently used in the diagnosis, management and monitoring of DES.5, 6, 7 In a recent review, 19 questionnaires were identified that may be used specifically for dry eye patients to assess symptoms, measure health-related quality of life (HRQL) and visual function. These questionnaires differ in their developmental methodology, design, and mode of administration in addition to the data collected and their psychometric properties. Specifically, some instruments retrieve dry eye symptomology and others take into consideration additional measures, including risk factors and HRQL.

The Ocular Surface Disease Index© (OSDI) (Allergan plc, Irvine, CA) is a validated questionnaire frequently used in clinical settings in addition to the research environment.7, 8 The questionnaire includes relatively few questions and is more easily applied than many longer, more complex questionnaires that are available.7, 8 The OSDI questionnaire has been shown to be a valid and reliable instrument in the assessment of dry eye severity. This 12-item questionnaire provides a rapid assessment of symptoms and HRQL and the questions are subdivided into three separate groups: ocular symptoms, vision-related functions and environmental factors. Patients are asked to recall the frequency of specific ocular symptoms and scenarios experienced “in the past week” along a five level Likert-type scale: “All of the time”, “Most of the time”, “Half of the time”, “Some of the time” and “None of the time”. An option of “Not applicable” is available for patients who did not perform the activities or experience any of the vision related functions and environmental conditions outlined in the questionnaire. The composite score ranges from 0 to 100, with higher scores representing greater severity.

The OSDI questionnaire was developed using several information sources, ranging from patient comments compiled during clinical studies to analyzing quality-of-life instruments. The initial item list was distributed to over 400 patients with DES to establish the frequency of symptoms or lifestyle restrictions experienced, in addition to using patient focus groups and clinician interviews. The preliminary 40-item OSDI questionnaire was reduced to the existing 12-item questionnaire after psychometric analysis examining the validity and reliability from two groups of patients with dry eye in a clinical trial.

Several steps of cognitive processing are utilized during the completion of questionnaires, including an understanding of the question, recalling the information required and communicating the response. Therefore, the mode of questionnaire delivery has been shown to affect the quality of the data obtained. Self-administered questionnaires require a greater amount of cognitive burden on the participant compared with face-to-face interviews with an examiner or researcher. However, the quality of data acquired from using questionnaire tools may vary with interviewer presence; while comprehension of the question can be improved, the questions that involve sensitive health- or lifestyle-related information may be under-reported when an examiner or researcher is present. These differences in individual interpretation could lead to additional variability in the responses between patients which will have an impact on the data. This has implications during the design of clinical studies and trials where the OSDI questionnaire may be used.

Based on previous interactions with participants, some common problems that arose during OSDI symptom testing was that participants did not know what certain words meant (e.g. gritty), and some did not understand the difference between blurry and poor vision. Since the OSDI is supposed to be provided to patients to complete without guidance, their own interpretation of questions may change over time and their understanding of the questions may not have any pertinence to dry eye disease at all. This would result in an inaccurate measurement of their dry eye symptoms. Therefore, the purpose of this study was to determine how providing a set of standardized instructions to participants during OSDI testing would affect outcome scores. Secondly, the visual function subsection (and possibly other items) of the OSDI may be impacted by presbyopia (e.g. items that query “reading” and “watching TV”). Therefore, this study also aimed to compare how standardized instructions impacted the OSDI scores in a younger versus an older population.

Methods

This study was designed and conducted in accordance with the ethical principles in the Declaration of Helsinki and received ethics clearance from the Office of Research Ethics at the University of Waterloo (Waterloo, Ontario, Canada). Informed consent was obtained from all participants prior to commencement of the study.

This was a prospective, non-randomized, deceptive study that was conducted over two consecutive days. To be eligible for the study, participants were required to be healthy (no active ocular disease, systemic conditions, or taking medications that could impact dry eye symptoms) non-contact lens wearers. In addition, participants must have scored at least 1 on the Subjective Evaluation of Symptoms of Dryness (SESoD) questionnaire. Participants were led to believe that the purpose of the study was to compare the OSDI questionnaire to the SESoD questionnaire.

It was not possible to predict how much of an effect providing instructions would have on the OSDI score. Therefore, this exploratory study was powered to detect a nominal 5.0 point difference in OSDI between the two methods of administration. Furthermore, a 10 point change in the OSDI represents an entire grade change (from mild to moderate, and from moderate to severe), to be able to detect a half-grade change is a reasonable target. To detect a 5.0 point change in OSDI (SD approximately 10) at α = 0.05 and 1 − β = 0.80, a minimum of 35 participants was required. Fifty young adult participants (aged 17–44 years, 38F, 12M), and 50 older participants (aged 45 years and above, 42F, 8M), were targeted for a total of 100 participants.

Eligible participants underwent a brief interview and general demographic information (age, sex) was documented. At their first visit, participants were asked to complete the OSDI questionnaire with no assistance provided other than the current instructions provided by Allergan. After completing the OSDI, participants had the opportunity to make specific comments on the clarity of the questions.

The second visit took place the day after the initial visit. During the second visit, participants confirmed any changes to their health and medications, and completed the SESoD questionnaire to ensure that symptoms did not significantly change from the previous days’ appointment. Participants were provided the OSDI questionnaire in exactly the same layout as the initial visit. The OSDI was then completed with specific assistance from a trained examiner. To avoid inter- and intra-examiner bias, a script was developed based on previous experience with the questionnaire and with consultation from the developer of the questionnaire and was intended to better explain how the questions may relate specifically to dry eye symptoms. The instructions that were read to the participants by the examiner (additional information provided to participants is bolded) are shown in Appendix 1. All participants were allowed to ask questions which were addressed by the examiner, as long as they fell within the confines of the script. After completing the OSDI, participants had the opportunity to comment on the clarity of the instructions, and provided feedback regarding the questionnaire.

Statistical analysis was conducted using GraphPad Prism 6 (GraphPad Software, Inc., La Jolla, CA, USA). Normal distribution testing was conducted using the Shapiro–Wilk test, with threshold set at α = 0.05. The mean difference between self-administration (SG) and examiner-guided administration (EG) was tested using the paired t-test for normal distributions, and the Wilcoxon Sign test in non-parametric distributions. Threshold for statistical significance was set at p < 0.05. This was also used to determine the mean differences for individual item scores between SG and EG.

Comparison between SG and EG administrations was examined with Bland–Altman plots. The correlation between SG and EG administrations was calculated using Spearman's rho.

Results

A total of 102 participants were enrolled in the study (80 females, 22 males) and one hundred completed the study. Of the two participants who did not complete the study, one participant was found to be ineligible (active ocular infection) and the other was lost to follow-up. As a requirement, all 100 participants who completed the study confirmed no changes to health or medications over the course of the study and did not have a significant change in their symptoms, as reported on the SESoD questionnaire, over the course of the two study days.

Method comparison of composite OSDI score

The OSDI score (mean ± SD) of the SG and EG administration was 32.0 ± 17.3 and 33.8 ± 19.6, respectively. The mean difference (mean ± SD) between SG and EG scores was 1.8 ± 11.4 (p = 0.34). A method comparison plot compares the SG to the EG administration (Fig. 1). The mean difference of SG versus EG is near zero, however ratings fell between ±22.4 of each other, 95% of the time. A scatter plot shows that the two methods of administrating the questionnaire were strongly correlated (Fig. 2) (Spearman's r = 0.81, p < 0.01).

Figure 1

A method comparison plot comparing self-guided and examiner-guided administration. The mean difference is 1.8 with coefficient of variability of ±22.4. Ratings fell between [−20.6, 24.2] 95% of the time.

Figure 2

Scatter plot showing correlation between self-guided and examiner-guided administration. The two methods were strongly correlated. Spearman's r = 0.81, p < 0.01.

Relationship between age and method of administration

The sample was further split into two age groups, under-45 (n = 50, age range 18–44), and 45-and-older (n = 50, age range 45–85). The under-45 age group showed a mean reduction of 1.2 points after EG, and the 45-and-older group showed a mean increase of 4.5 points. None of these differences were significant (p > 0.05). Also, while the mean difference in SG and EG for both groups were also small, the standard deviation of the mean differences was twice as high in the 45-and-older group as it is in the under-45 group. Table 1 summarizes this finding.

Table 1

Mean shift (±SD) in scores for both age groups. None of the changes are statistically significant.

	Under-45 (n = 50)		45-and-older (n = 50)
	Self-guided	Examiner-guided	Self-guided	Examiner-guided
OSDI score	32.8 ± 15.3	31.5 ± 14.1	31.4 ± 19.0	35.9 ± 23.7
Mean difference ± SD	−1.2 ± 7.3, p = 0.39		4.5 ± 14.2, p = 0.07

This is reflected in the method comparison plots in Fig. 3, where the 95% interval is wider in the 45-and-older than the under-45 group. Two scatter plots show the correlation of the two methods between the two age groups (Fig. 4). While both groups each showed strong correlation between the two methods (Spearman's r = 0.88, p < 0.01, and Spearman's r = 0.76, p < 0.01, respectively), the 45-and-older age group showed more variability.

Figure 3

Method comparison plots comparing self-guided and examiner-guided administration in left: under-45 age group, right: 45-and-older age group. The coefficient of variability is 14.3 for the under-45 group, and 27.8 for the 45-and-older group.

Figure 4

Scatter plot showing the difference in variability between under-45 and 45-and-older age groups. Both groups show strong correlation (Spearman's r = 0.88, p < 0.01 and Spearman's r = 0.76, p < 0.01, respectively), however the under-45 group showed less variability than the 45-and-older group.

Clarity of questions

Without examiner guidance, the questions that participants had the least trouble with pertained to “Reading”, “Watching TV”, “Windy conditions”, and “Areas that are air conditioned”. For each of those questions, only 2 out of 100 participants responded that they were not clear or easy to understand. In contrast, 22 out of 100 participants responded that “Poor vision” was not clear or easy to understand and 11 out of 100 participants responded that “Blurry vision” and “Painful/sore eyes” were not clear or easy to understand.

Following the EG administration, 92, 89 and 72 out of 100 participants found that the explanations were helpful to clarify “Blurry vision”, “Poor vision” and “Painful/sore eyes”, respectively. Table 2 summarizes participant responses to clarity of questions.

Table 2

Summary of participant responses to questions.

Questions	Self-administration (n = 100)		Examiner-guided (n = 100)
	Did you find the questions clear and easy to understand?		Was the explanation provided for each question helpful/unnecessary/unhelpful?
	No. of participants responding “Yes”	No. of participants responding “No”	No. of participants responding “helpful”	No. of participants responding “unnecessary”	No. of participants responding “unhelpful”
Eyes that are sensitive to light?	90	10	85	13	2
Painful/sore eyes?	89	11	72	27	1
Eyes that feel gritty?	94	6	81	18	1
Blurry vision?	89	11	92	7	1
Poor vision?	78	22	89	10	1
Reading?	98	2	77	22	1
Driving at night?	97	3	74	25	1
Working with a computer/bank machine?	96	4	72	26	2
Watching TV?	98	2	73	26	1
Windy conditions?	98	2	76	22	2
Places or areas with low humidity?	96	4	82	16	2
Areas that are air conditioned?	98	2	77	21	1

Impact of method of administration on individual item score

While most of the items showed a statistically significant change after providing instructions, the change (whether it was an increase or decrease) was relatively inconsequential and none of the changes were considered to be clinically significant. This suggests that with respect to scoring, EG is neither better nor worse than SG for administering the OSDI. Table 3 shows the shift in scores for each item.

Table 3

Mean difference (±SD) between SG and EG for each question item.

Questions	Self-examiner guided	p value
Eyes that are sensitive to light?	−0.6 ± 1.1	<0.01*
Painful/sore eyes?	0.5 ± 0.8	<0.01*
Eyes that feel gritty?	0.2 ± 0.8	<0.01*
Blurry vision?	0.1 ± 0.9	0.14
Poor vision?	0.1 ± 1.0	0.23
Reading?	0.3 ± 0.9	<0.01*
Driving at night?	0.1 ± 0.8	0.16
Working with a computer/bank machine?	0.2 ± 0.8	0.01*
Watching TV?	0.2 ± 0.9	0.01*
Windy conditions?	−0.1 ± 0.8	0.36
Places or areas with low humidity?	−0.3 ± 1.0	<0.01*
Areas that are air conditioned?	−0.2 ± 0.9	0.01*

p < 0.05.

Discussion

In a previous study, differences were found in scores obtained using the McGill pain questionnaire self-administered by pen-and-paper and by interview. These findings indicate the importance of keeping the mode of questionnaire administration consistent, specifically for questionnaires that may be used as an outcome variable in clinical studies. This also has an impact upon the sensitivity of the questionnaire in identifying the severity of disease state within a clinical and experimental setting. According to the administration and scoring manual, the OSDI questionnaire should be completed by the patient prior to seeing the physician, to ensure that the feedback from the physician does not influence the patient responses to the questionnaire. If the patient seeks clarification on a particular question, it is indicated that the question should be read to the subject verbatim and the patient should be encouraged to use his or her own interpretation of the question. However, certain questions, including sensitivity to light, blurry vision, and poor vision, may not necessarily relate to dry eye alone. These symptoms may be precipitated by other ocular conditions (e.g. glaucoma, convergence insufficiency). By providing instructions on how to complete the questionnaire, it may help guide the patient's interpretation of the questions.

A potential concern related to the design of this study is that the order of administration presented was not randomized. Unlike the between-subjects study design used by Klepac et al., this study examined within-subject differences. The strength of using a within-subject design is that it controls for individual participant factors. The disadvantage however is that using a within-subject design also made it impossible to implement group randomization. Essentially, participants who completed the OSDI with guidance and instructions on the first day would surely remember them as they are completing the OSDI without guidance on the second day. In an attempt to reduce any order effect, participants were informed through deception that the purpose of the study was to compare the OSDI to the SESoD. Despite this, however, it is possible that the lack of randomization could have influenced the results.

Both methods of administration showed a strong correlation with each other, suggesting that the EG method followed a similar behaviour as the SG method. The method comparison plots showed that the mean difference between the two methods was very small. Therefore on average, there was no tendency for one method to rate significantly higher or lower than the other method. There is however a large 95% agreement interval (approximately between −20 and +20) between SG and EG methods of administration. The small mean difference is due to the large variability averaging itself out.

When the sample population was broken down into under-45 and 45-and-older age groups, there was a difference in response between the groups. The mean difference in both groups was still approximately equal, but the 95% limits of agreement were different. The 95% interval is wider for the 45-and-older age group than for the under-45 age group, suggesting that providing instructions to the 45-and-older age group had a larger impact on their answers than the under-45 age group. However, since there was no objective clinical testing involved, this study could not provide an explanation as to why the 45-and-older age group had a larger variation between the two methods of administration. The visual function subsection scores of the OSDI may have been impacted in the older subgroup by non-dry eye conditions such as presbyopia (blurry/poor vision) or cataracts (sensitivity to light), which are characteristic of the ageing population.14, 15 This hypothesis could be tested in future evaluations.

Despite the fact that many participants reported that instructions helped clear up questions that queried “poor vision”, “blurry vision”, and “painful/sore eye”, there was no statistical difference in their scores for those questions, except for the question that queried “painful/sore eye”. Similarly, most participants reported that the questions that queried reading and watching TV were the most easy to understand, yet there was a statistically significant difference in their scores after being given instructions. This suggests that the instructions may have changed their perception on what reading and watching TV meant. Their original interpretation of the question may also have been related to whether or not they had difficulties with or without glasses to read or see the TV. Overall these changes were less than 0.62 units and are not considered to be clinically significant. While there was not a significant difference between methods of administration overall, up to 22% of participants reported that certain questions were confusing, and a large portion of participants (72–92%) found that explanations were helpful. This could possibly suggest that providing instructions to patients is helpful, but since the data does not appear to show any clinically significant changes, we are unsure to what extent this benefit holds. One reason why explanations may have appeared helpful to participants but did not change the overall outcome may be related to the nature of the EG administration. The EG administration was not validated and its performance was never tested. With further refinement, it may be possible to determine whether EG administration could be more helpful. It is also important to note that since the EG administration was not validated, providing instructions to participants should be avoided when using the OSDI in clinical trials.

Conclusion

Overall there was no significant difference in OSDI outcomes when participants completed it with or without examiner guidance, however the OSDI scores from the older population appears to be affected more with examiner guidance. Since some patients may require clarification on certain questions, and a majority of them have expressed that explanations have helped their understanding of the questions, it is recommended that the OSDI be administered with examiner guidance. Additionally, by explaining questions on the OSDI it may help to educate patients that the OSDI (or other symptom assessments) is a form of dry eye management.

Funding

Funding for this study was provided by Allergan plc.

Conflict of interest

Over the past three years, the Centre for Contact Lens Research have received research funding and/or honoraria from the following companies: Advanced Vision Research, Alcon, AlgiPharma, Allergan, Contamac US, CooperVision, Essilor, Johnson & Johnson Vision Care, Ocular Dynamics, OCULUS, TearScience.