Validation of a glaucoma knowledge assessment in glaucoma patients.

SUMMARY: To develop metrics to identify knowledge deficits and barriers to learning in glaucoma patients, this study seeks to validate a glaucoma knowledge assessment to use in the evaluation of knowledge in glaucoma patients.
PURPOSE: Glaucoma treatment adherence appears to improve when patients demonstrate a greater knowledge of the disease and its treatment. This study seeks to validate a glaucoma knowledge assessment in a glaucoma patient population to assist in patient educational assessments and interventions. PATIENTS AND METHODS: The National Eye Health Education Program's (NEHEP) glaucoma knowledge assessment has previously been suggested as a useful measure to assess glaucoma patient's knowledge. This questionnaire was administered in glaucoma patients along with a more comprehensive 49-question examination. Demographic data and health literacy were assessed. Statistical analyses were performed to assess the validity of the assessments.
RESULTS: A total of 12 glaucoma patients completed the knowledge assessments. The mean ± standard deviation age of the patients was 69±14 years, and the duration of glaucoma was 14±13 years. The participants' mean score on the NEHEP assessment was 7.3±0.8 (of 10, 73% correct) vs 29.3±7.3 (of 49, 60% correct) on the comprehensive assessment. The value of coefficient α was 0.592 for NEHEP and 0.872 for the cumulative assessment. The P-value (proportion of examinees answering correctly) ranged from 0 to 1 for individual questions. Item point-biserial correlation values for each question ranged from -0.402 to 0.813. DISCUSSION: The NEHEP quiz may be a good starting point for the development of a reliable knowledge assessment tool to measure and monitor glaucoma knowledge, due to its concise nature and reasonable level of difficulty. This study suggests that questions included in the currently available questionnaires vary widely in difficulty and ability to differentiate knowledge level, which may ultimately compromise reliability and utility of existing examinations. Further examination of individual questions and teaching points included in the current assessments may help to construct increasingly reliable and useful knowledge assessments in the future.

Introduction

Glaucoma is a chronic, progressive blinding disease that typically requires lifelong treatment and follow-up care. Despite the lack of symptoms in early and moderate disease, it is key for patients to adhere to treatment plans, including physician visits and treatments to prevent vision loss.1 This can be especially challenging, considering the time burden of physician visits and potential side effects from medical and surgical treatments. Adherence to medical therapy in glaucoma, like other chronic diseases, has been documented to be poor.2,3 Up to 80% of glaucoma patients on medications have been shown to have adherence issues,4 and issues with non-adherence appear to be worse in patients on multiple medications.5,6

The current literature suggests that glaucoma patient adherence is improved with better patient education.7 Greater knowledge of the disease process may allow patients to better understand their treatment plan and actively participate in their care. Surveys of glaucoma patients indicate that there is room for improvement in patient education; specifically, patients have demonstrated lack of knowledge regarding the disease process in glaucoma, medication side effects, and the natural history of glaucoma without treatment.8 A lack of understanding of the severity of glaucoma has been correlated with poor adherence.9

Considering these difficulties, it can be inferred that improved patient education, leading to improved understanding of glaucoma, might improve adherence to treatment and, ultimately, lead to better outcomes. In order to better direct patient education, it might be useful to have a method to formally assess glaucoma knowledge during routine clinical care. This could be useful to assess both baseline knowledge and the effectiveness of educational interventions. We previously identified four existing glaucoma knowledge assessment quizzes available to the public:10 the National Eye Health Education Program (NEHEP) Eye-Q test,11 the Prevent Blindness America (PBA) Eye-Q test,12 Gray Glaucoma Knowledge Assessment,13 and Hoevenaars Glaucoma Knowledge Assessment.14 The Gray and Hoevenaars assessments were developed as research instruments. In that study, glaucoma clinicians were surveyed on the quality of the four different assessments and determined a preference for the 10-question NEHEP Eye-Q test for use in clinical practice.

Here, we seek to validate this questionnaire in glaucoma patients and determine the appropriateness of its use in assessing knowledge of glaucoma in these patients. The results of this quiz are compared to the results of a 49-question cumulative examination comprising all questions from the abovementioned four knowledge assessments.

Patients and methods

This prospective study was approved by the Duke University Medical Center’s institutional review board and was conducted in accordance with the standards of Health Insurance Portability and Accountability Act and the Declaration of Helsinki.

The participants were recruited from the Glaucoma Service of a tertiary care academic practice. Written informed consent was obtained from all participants. The inclusion criteria were: capable and willing to provide consent, history of diagnosed glaucoma or history of ocular hypotensive therapy (eg, high-risk glaucoma suspects or ocular hypertensives) based on glaucoma specialist’s documented assessment in the medical record, at least 18 years of age, English speaking (the glaucoma knowledge assessments were in English), and willingness to complete the study activities. Notably, there were no minimum criteria for visual acuity. If the study participant was unable to see to complete an activity (eg, unable to complete the literacy instrument), the study staff read the question or item aloud to the study participant. No compensation was offered for this study.

The potential participants were approached by the study team after an introduction of the study by a member of their care team (per institutional policy). The participants completed the study activities while waiting for their glaucoma doctor during a clinic visit; the study activities included the completion of two glaucoma quizzes, a demographic questionnaire, and a health literacy assessment (the Rapid Estimate of Adult Literacy in Medicine – Short Form (REALM-SF) instrument). The demographic and educational information collected included age, race/ethnicity, gender, education level, glaucoma treatments received (medications, laser, surgery), and presence of any other eye disease. The medical record was reviewed to confirm the information provided by the study participants. The glaucoma quizzes consisted of: 1) the 10-question NEHEP Eye-Q test and 2) a 49-question quiz consisting of questions (in random order) from the NEHEP Glaucoma Eye-Q test, the PBA Glaucoma Eye-Q test, the Gray Glaucoma Knowledge Assessment, and the Hoevenaars Glaucoma Knowledge Assessment. The format of each assessment was as follows: NEHEP, 10 true–false questions; PBA, 9 true–false questions; Gray, 11 short-answer questions; and Hoevenaars, 19 true–false questions.

Statistical analyses of the quiz questions were performed using the Iteman 4 (Assessment Systems) validation software: 1) the mean scores, the alpha value, and the standard error of measurement (SEM) were calculated for the NEHEP quiz, the 49-question quiz (total), and the subtotals of the PBA, NEHEP, Gray, and Hoevenaars questions within the 49-question quiz; 2) intercorrelation values were calculated for PBA, NEHEP, Gray, and Hoevenaars subtotals; and 3) the P-value (the proportion of participants who answered the question correctly, evaluating how easy or hard a question is) and the point–biserial correlation value rpbis (which correlates the response with the total score, evaluating how well each question correlated to the overall score) were calculated for each individual question. Descriptive statistics were used for the demographic information of the participants.

Results

A total of twelve glaucoma patients were recruited in this study. The mean age of the patients was 69±14 years, and 58% (n=7) of patients were female (Table 1). The patients had been diagnosed with glaucoma for a mean of 14.3±12.7 years, and most (92%, n=11) were taking glaucoma medications. The majority of patients had undergone glaucoma surgery (83%, n=10). The educational level ranged from no high school education to graduate degree; 25% of patients (n=3) had less than a high school diploma and 50% (n=6) had at least some college education. The mean REALM-SF score to assess health literacy was 5.5±2.1, which is consistent with a seventh to eighth grade level of education.15

Table 1

Demographics of participants

Characteristic	N=12
Age, mean (SD); range	69 years (14); 42–88 years
Gender	5 males7 females
Race	4 whites8 blacks
Ocular comorbidities
Age-related macular degeneration	2
Cataracts	7
Diabetic retinopathy	2
History of eye surgery (non-glaucoma)	9
Glaucoma history
Number of years with glaucoma, mean (SD); range	14.3 years (12.7); 1–40 years
Number of patients on glaucoma medications	11
History of glaucoma laser	9
History of glaucoma surgery	10
Level of education
No high school	1
Some high school	2
High school graduate or GED	3
Some college	4
College degree	1
Graduate or professional degree	1
REALM-SF score, mean (SD)	5.5 (2.1); scores range from 1 to 7

Abbreviations: SD, standard deviation; GED, General Educational Development test; REALM-SF, Rapid Estimate of Adult Literacy in Medicine – Short Form.

The highest mean score of knowledge assessments was for the NEHEP assessment (73%), followed by the PBA (67%), Hoevenaars (64%), and Gray (35%; Table 2). The cumulative assessment of all quizzes had a mean score of 60%. The value of coefficient α was 0.592 for NEHEP and 0.872 for cumulative assessment; the SEM for the NEHEP quiz was 1.1, less than that for the cumulative quiz (2.6). The P-value (proportion of examinees answering correctly) ranged from 0 to 1 for all questions. Item point–biserial correlation values for individual questions ranged from −0.402 to 0.813. The PBA assessment correlated most with the other assessments (Table 3) with NEHEP second in quiz intercorrelation.

Table 2

Summary of participant scores and reliability

Measure	Part 1 (NEHEP Eye-Q test)	Part 2 total	Part 2 domains
			Gray	Hoevenaars	NEHEP	PBA Eye-Q test
Number of questions	10	49	11	19	10	9
Mean score (%, SD)	7.3 (73, 1.8)	29.3 (60, 7.3)	3.8 (35, 2.9)	12.2 (64, 2.3)	7.3 (73, 1.5)	6.0 (67, 1.8)
Minimum score, maximum score	5, 10	19, 43	0, 9	8, 17	5, 10	4, 9
Alpha	0.592	0.872	0.835	0.592	0.416	0.478
SEM	1.134	2.599	1.199	1.47	1.181	1.27

Notes: Gray, knowledge questions from Gray et al;13 Hoevenaars, knowledge questions from Hoevenaars et al.14

Abbreviations: NEHEP, National Eye Health Education Program; PBA, Prevent Blindness America; SD, standard deviation; SEM, standard error of measurement.

Table 3

Quiz intercorrelation

Domain	PBA Eye-Q test	NEHEP Eye-Q test	Hoevenaars	Gray
Gray	0.789	0.589	0.355	1
Hoevenaars	0.607	0.774	1
NEHEP	0.837	1
PBA	1

Notes: Hoevenaars, knowledge questions from Hoevenaars et al;14 Gray, knowledge questions from Gray et al.13

Abbreviations: PBA, Prevent Blindness America; NEHEP, National Eye Health Education Program.

The P-value (proportion of patients answering each question correctly) for each question is presented in Table 4. Notably, the Gray and Hoevenaars assessments were the only two assessments with “hard” questions (P<0.334). “Easy” questions (P>0.667) were present in each of the four assessments. There was no particular subject category of question that was consistently either “easy” or “hard” when examining the proportion of patients answering the questions correctly.

Table 4

Analysis of individual questions

Question	P-valuea	rpbisb	Question category
“Easy” questions (items with P>0.667)
Hoevenaars Question 6	1	0	Vision loss
Hoevenaars Question 7	1	0	Causes and types
Hoevenaars Question 19	1	0	Systemic disease
Hoevenaars Question 14	1	0	Systemic disease
Hoevenaars Question 8	1	0	Symptoms
NEHEP Eye-Q test Question 6	1	0	Clinical course
NEHEP Eye-Q test Question 3	1	0	Symptoms
NEHEP Eye-Q test Question 7	1	0	Eye pressure
Hoevenaars Question 18	0.917	0.11	Eye pressure
Gray Question 5	0.833	0.342	Risk factor, family history
Hoevenaars Question 11	0.833	0.101	Risk factor, family history
NEHEP Eye-Q test Question 10	0.833	0.307	Diagnosis
PBA Eye-Q test Question 8	0.833	0.412	Clinical course
PBA Eye-Q test Question 9	0.833	0.273	Diagnosis
PBA Eye-Q test Question 2	0.833	0.101	Risk factor, family history
NEHEP Eye-Q test Question 2	0.75	0.347	Risk factor, family history
NEHEP Eye-Q test Question 8	0.75	0.469	Vision loss
PBA Eye-Q test Question 4	0.75	0.168	Prevalence
“Medium” questions (items with 0.334<P<0.667)
Hoevenaars Question 4	0.667	0.657	Causes and types
Hoevenaars Question 5	0.583	0.332	Eye pressure
PBA Eye-Q test Question 1	0.583	0.603	Symptoms
Gray Question 1	0.5	0.574	Causes and types
Gray Question 3	0.5	0.178	Vision loss
Hoevenaars Question 1	0.5	0.36	Vision loss
NEHEP Eye-Q test Question 5	0.5	0.574	Symptoms
PBA Eye-Q test Question 5	0.5	0.334	Risk factor, age
PBA Eye-Q test Question 6	0.5	0.44	Risk factor, race
Gray Question 4	0.417	0.818	Vision loss
Gray Question 7	0.417	0.705	Eye drops
Hoevenaars Question 16	0.417	0.705	Risk factor, race
NEHEP Eye-Q test Question 9	0.417	0.705	Eye pressure
NEHEP Eye-Q test Question 1	0.417	0.649	Risk factor, race
PBA Eye-Q test Question 7	0.417	0.79	Diagnosis
“Hard” questions (items with P<0.334)
Gray Question 2	0.333	0.438	Causes and types
Gray Question 11	0.333	0.843	Clinical course
Gray Question 8	0.333	0.754	Eye drops
Hoevenaars Question 10	0.333	0.189	Systemic disease
Hoevenaars Question 9	0.25	0.53	Clinical course
Hoevenaars Question 18	0.167	0.261	Eye pressure
Gray Question 6	0.083	0.356	Clinical course
Hoevenaars Question 3	0.083	0.356	Vision loss
Items that poorly differentiate examinees (items with rpbis <0 or P=0)
NEHEP Eye-Q test Question 4	0.583	−0.402	Risk factor, age
Hoevenaars Question 15	0.083	−0.191	Vision loss
Gray Question 10	0.083	−0.146	Eye drops
Hoevenaars Question 2	0.917	−0.072	Risk factor, age
Hoevenaars Question 12	0.75	−0.036	Eye pressure
Hoevenaars Question 17	0.75	−0.036	Risk factor, refractive error
PBA Eye-Q test Question 3	0.75	−0.036	Causes and types
Gray Question 9	0	0	Eye drops

Notes:

P-value, proportion of examinees who answered the item correctly;

rpbis, point–biserial correlation value of response with total score; Hoevenaars, knowledge questions from Hoevenaars et al;14 Gray, knowledge questions from Gray et al.13

Abbreviations: NEHEP, National Eye Health Education Program; PBA, Prevent Blindness America.

Discussion

In order to better assess glaucoma patient’s knowledge and monitor learning during routine clinical care, a standardized glaucoma knowledge assessment would be instrumental. This would be especially useful in light of literature that supports that adherence in glaucoma patients is often poor4 and tends to improve with better knowledge of the disease.7 The purpose of this study is to assess the validity of commonly available glaucoma knowledge assessments in an effort to identify useful means to evaluate patient’s knowledge in a busy clinical setting. As our recent report suggested the NEHEP glaucoma knowledge assessment as the most useful of these quizzes in this setting,10 we focused on this assessment in this study.

In this group of glaucoma patients, the mean score on the NEHEP knowledge assessment was 7.3±0.8 (of 10). This mean score may suggest an appropriate level of difficulty for most patients. However, the value of coefficient α for the NEHEP quiz was 0.5, suggesting that the quiz may have poor internal consistency. The value of coefficient α for the cumulative 49-question quiz was more robust at 0.8, which suggests that the more comprehensive quiz might be more reliable and internally consistent for determining patient’s knowledge. Interestingly, the Gray assessment also had a high α value, which could be attributed to its short answer, more interactive nature. Despite the high α coefficient value for the Gray assessment, it should be noted that multiple short-answer questions were either omitted or inappropriately answered with a single word by patients in this study. It is not clear if these questions were omitted due to lack of knowledge or due to the extra effort the short-answer question required over the true–false questions (all true–false questions were completed appropriately in the study). This may suggest that although short–answer questions may be more revealing about patient’s knowledge, they may not be the most efficient and effective means to assess knowledge in a busy clinic setting, as suggested by the glaucoma specialists in our prior study.

Furthermore, when examining the item point–biserial correlation, which serves as an indicator of differentiating power for individual questions, this varied widely from −0.402 to 0.813 (possible range from −1.0 to 1.0, with higher values suggesting greater discriminating ability). This suggests that some questions used in each questionnaire would have limited utility in determining the level of glaucoma patient’s knowledge. For example, 3 of 10 questions in the NEHEP quiz had an item point–biserial value of 0.0, suggesting no ability to discriminate between low-knowledge and high-knowledge assessment takers.

The results of this study suggest that the longer, cumulative knowledge assessment may provide a more reliable indicator of glaucoma patient’s knowledge. Although intuitively it is reasonable to assume a more thorough test would provide more reliable results, it would also be pragmatic to identify a concise means to assess patient’s knowledge during a busy clinical encounter. For example, the three NEHEP questions that received a 100% correct score and the item point–biserial value of 0.0 appear to add little to the differentiation of higher and lower knowledge patients in this study. These questions may still have a role in educating the patient, for example, to reinforce important information.

The strength of this study includes the examination of four commonly available knowledge assessments in a single patient population. The patient population in this study had a diverse range of ages (44–88 years), an equal distribution of genders, a wide range of time since glaucoma diagnosis (1–40 years), and a diverse level of education (from no high school to graduate degree). The questions included in this study consisted of a wide range of topics, and each of the 49 questions was individually analyzed for its appropriateness. The weaknesses of this study are the small sample size and the overall high level of glaucoma severity, considering 10 of the 12 patients have undergone glaucoma surgery, which may limit its generalizability. There was no minimum number for assessing validity; thus, a small group of diverse patients was used to preserve the larger population of quiz-naïve potential participants in our patient pool for future studies; the limited sample size likely contributed to the lower alpha values and larger SEM values. Additionally, there was no known “gold standard” for glaucoma knowledge; hence, the topics covered in this study may not be most relevant to all patients and all clinicians. Additional limitations include completion at a single institution and only English-speaking patients were included.

Overall, this study aims to assist in the educational initiatives of glaucoma patients by assessing the validity of currently available glaucoma knowledge assessments. The NEHEP knowledge assessment, previously ranked by glaucoma clinicians as most useful for clinical practice, here was suggested to have a reasonable level of difficulty but a potential lack of internal consistency. Approximately 50 glaucoma knowledge questions assessed here showed varying levels of difficulty and internal consistency.

Clinicians can use the data on each of the questions as a starting point to create their own assessments for their patients, potentially allowing glaucoma knowledge assessments to be tailored to individual patient and provider needs. In the future, it might be useful to tailor not only a general glaucoma quiz but also assessments targeted for patients with varying educational backgrounds and/or glaucoma severity. By developing more accurate means to assess glaucoma knowledge, we ultimately hope to better implement and assess educational interventions for our patients.