Development and validation of a questionnaire to self-assess patient knowledge of direct oral anticoagulants (KODOA-test).

OBJECTIVE: Few studies have examined associations between patient knowledge of direct oral anticoagulants (DOAC) and clinical outcomes, mostly because of the lack of validated questionnaires for assessing knowledge. The aim of this study was to develop and validate a questionnaire to self-assess knowledge of DOAC.
METHODS: Twelve anticoagulation experts participated in the questionnaire development process to ensure content validity. The Knowledge Of Direct Oral Anticoagulants (KODOA)-test was submitted to patients on DOAC and to pharmacists to assess construct validity. Responsiveness was evaluated after educational counseling. Test-retest reliability was assessed to ensure stability over time, and Cronbach's α was calculated for internal reliability. Index of difficulty and item discrimination (D-value) were calculated to assess the performance of single items.
RESULTS: The KODOA-test contains 15 items with multiple-choice answers. Each correct answer scores 1 point (max. score of 15). The KODOA-test was administered to 32 patients on DOAC and 28 pharmacists. Pharmacists scored significantly higher than patients at baseline (median score 13.3 vs 10.0; p<0.001), supporting construct validity. Patient scores increased significantly after educational counseling (median score 11 [interquartile range 2] vs 14 [interquartile range 3]; p<0.001). Test-retest and Cronbach's α were acceptable with a Pearson's correlation of 0.8 and an α of 0.67. The index of difficulty for most items was satisfactory (0.38-0.72) and the mean D-value was 42.5%.
CONCLUSION: The KODOA-test is a brief, valid, and reliable knowledge self-assessment questionnaire that may be used in clinical trials to investigate associations between knowledge increase and patient-related outcomes.

Introduction

After a 50 years use of vitamin K antagonists (VKAs) as oral anticoagulant agents (OACs), a new class of substances with different mechanism of action, the direct oral anticoagulants (DOAC), also called non-vitamin K oral anticoagulants, has been developed. DOAC are recommended for the prevention and treatment of thromboembolic diseases such as for the long-term prevention or treatment of thrombosis1 and for the prophylaxis of stroke and systemic embolism in nonvalvular atrial fibrillation (AF).2 Controlling blood values is needed to maintain appropriate anticoagulation with VKAs due to the narrow therapeutic window, which makes the treatment challenging. In contrast to VKA, DOAC do not require routine monitoring, have a fixed dose regimen, and possess no restriction on dietary consumption of vitamin K-containing food. Therefore, DOAC seem more convenient for patients than therapy with VKAs. In practice, use of DOAC to treat AF is increasing.3,4 However, due to their short half-life, DOAC’s anticoagulant effect is likely to be rapidly reduced after the omission of one dose when no counterbalancing action is undertaken.5 Thus, DOAC’s daily intake requires a strict timing adherence to ensure appropriate therapeutic coverage. Because DOAC do not require consultations for monitoring, the opportunity to discuss aspects of adherence with the patient during such encounters will be missing. Consequently, other ways of insuring adherence to DOAC are required.

Improvement of knowledge and adherence in patients on OACs, in particular DOAC, is needed.6 For VKAs, enhanced patient knowledge of OACs and the underlying disease was associated with improved long-term adherence or better therapeutic outcomes.7–9 To study associations between knowledge and adherence, validated questionnaires are needed to assess patient knowledge of DOAC. To date, various questionnaires exist that assess OAC knowledge. However, only a few have been psychometrically validated, such as the Oral Anticoagulation Knowledge (OAK),10 the Anticoagulation Knowledge Assessment (AKA) test11 and the recently published Anticoagulation Knowledge Tool (AKT).12 Only AKT can be used for VKAs and DOAC. To our knowledge, there is no specific knowledge assessment questionnaire for DOAC available.

The objective of this study was to develop and validate a questionnaire to self-assess patient knowledge of DOAC.

Methods

Trial design

The study was initiated by the Pharmaceutical Care Research Group of the University of Basel. The study was conducted in accordance with the Declaration of Helsinki and has been registered at ClinicalTrials.gov ID NCT03124654. The study was approved by the regional ethic committee (Ethikkom-mission Nordwest-und Zentralschweiz UBE15/126). All patients gave written informed consent.

Development of the Knowledge Of Direct Oral Anticoagulants (KODOA)-test

Literature search

A systematic literature search in Medline and Embase was conducted in March 2015 to retrieve published questions assessing patient knowledge of anticoagulation treatment. The search strategy was performed with truncated terms: “Patient* education*” AND “Anticoagulant*” OR “ Anticoagulation treatment*” OR “Knowledge” and the limits: English language, human, published from 2005 to march 2015. Abstracts were screened and full texts were retrieved. Two authors (CM and VA) extracted the items of interest. Items regarding VKAs were adapted to DOAC if applicable. Consensus was reached by discussion. Swiss summaries of Product Characteristics, the Updated European Heart Rhythm Association (EHRA) Practical Guide on the use of non-vitamin K antagonist anticoagulants,13 the patient guide for taking DOAC from a cardiology patient page14 and patient information leaflets of the 4 in Switzerland authorized DOAC were used to retrieve additional content. Items were grouped into the 9 educational topics as follows:15 1) underlying disease; 2) risk–benefit of treatment; 3) mode of action; 4) application and treatment adherence; 5) accessing health care providers; 6) relevant blood tests; 7) medication interactions; 8) diet and lifestyle; and 9) self-care.

Item reduction and content validity

A panel of experts was created with 12 health care professionals (HCP; 4 nurses, 4 pharmacists, and 4 physicians) who had experience with DOAC patients. They selected the relevant items for patient knowledge of DOAC and determined the extent to which the items represent the construct of interest (content validity).16

The item reduction occurred with an online survey (2 rounds) followed by a focus group discussion. The 9 educational topics (first round) and the single items (second round) were ranked in descending order (first = most relevant topic, last = most irrelevant topic). The focus group discussion aimed at defining which items had to be included in the questionnaire by defining a threshold above which the items were perceived as relevant for patient knowledge. Discussion and setting threshold occurred by voting until a consensus was found (unanimity). The session was recorded and transcribed verbatim.

Answer format

Answers to the items were developed as multiple-choice format because this is the most appropriate and efficient form for assessing cognitive knowledge.17 One correct and 2 incorrect response alternatives were adapted from the original article when available or were newly created. A score was defined as 1 for a correct response and 0 for an incorrect or no response. Confidence about the given response was assessed with a single question with a yes/no answer option.

Educational manual with correct answers as counseling guide

A manual with referenced correct answers and background information about the reason for correctness was generated in order to standardize the education provided to patients. The manual is not supposed to be given to patients. The manual starts with an instruction section about how to dispense the educational counseling. The HCP is invited to discuss systematically each question with the patient. Whenever the patient gave a wrong answer or ticked the “to be uncertain about the question” option, HCPs are instructed to give the correct answer with background information according to the manual. Then HCPs should ask if the issue is clear or if the patient needs further information. When every question of the KODOA-test has been discussed as described, HCPs should ask the patient if there are any unanswered questions about their DOAC. Although not part of the questionnaire, information about intake time of DOAC was added to cover all instructions needed to counsel. The manual was used by two investigators (CM and VA) during counseling sessions.

Pilot testing

Comprehension and readability of the finalized KODOA-test was presented to the experts who participated in the focus group discussion and 5 patients who were not included in the validation study. They rated structure, content, comprehension, and response alternatives by writing commentaries in free text. Adaptions were made accordingly.

Validation of the KODOA-test

Patient eligibility

Patients were eligible if they filled a prescription for DOAC (rivaroxaban, edoxaban, dabigatran, or apixaban) in a community pharmacy or if a DOAC was present in their medical record, if they suffered from AF, deep vein thrombosis, or pulmonary embolism, if they were ≥18 years old, and if they were able to give written informed consent in German. Patients with an orthopedic indication for DOAC, with dementia, or unable to provide written and/or oral understanding in German were excluded from the study.

Data collection

A selection of community pharmacists engaged as experts for the state exams of the University of Basel were invited to participate in the research and asked to distribute this invitation to all community pharmacists with whom they were working. Three dedicated community pharmacies located in Basel City and Münchenstein accepted to serve as recruiting places. Pharmacy staff asked patients with a prescription for DOAC to participate in the study and handed out the patient information and the informed consent forms. Patients who agreed to participate fixed 2 appointments within 2 weeks at the community pharmacy.

Patients filled in the KODOA-test 4 times. During the first appointment at the community pharmacy (T1), feasibility and acceptability were also assessed. Immediately after patients filled in the KODOA-test, they rated on a 4-point Likert scale structure, readability and time to fill in the questionnaire, comprehensibility and ambiguity of the items, and the available answer options.

During the second appointment at the community pharmacy, patients filled in the KODOA-test again (T2) and received educational counseling according to the wrong given answers. Immediately after, patients filled the KODOA-test a further time (T2 + edu). Test–retest was assessed with answers from T1 and T2, and responsiveness with answers from T2 and T2 + edu.

Approximately 2 weeks after T2, patients received the KODOA-test by post (with a paid reply envelope) and filled it at home (T3) to assess sustainability of the educational counseling. Patient follow-up by the treating physician (general practitioner or cardiologist) took place as usual.

Patient characteristics were obtained through written survey at T1 and included demographic (age, gender, educational background, and duration of DOAC therapy) characteristics and 3 health literacy questions: difficulties to understand written information about medication (yes/neutral/no); difficulties to understand verbal information about medication (yes/neutral/no); confidence when filling out forms (yes/neutral/no). One question assessed self-estimated knowledge of DOAC with the answer options bad, moderate, good, or excellent. At T3, patients were asked to rate following statement on a five-point Likert scale: “After educational counseling I know more about my DOAC” (1= I do agree; 5= I do not agree).

A personal letter was sent to 29 pharmacists asking them to answer the KODOA-test without consulting additional media and to return it using a prepaid reply envelope.

Statistical methods

Where appropriate, frequencies, mean, and standard deviations; or median and interquartile ranges (IQR) are presented. Kruskal–Wallis test was used to compare numerical variables between 4 groups. All data were entered and analyzed using SPSS statistical package version 24.0 (IBM Corporation, Armonk, NY, USA), and p-values <0.05 were considered significant.

Validation parameters

Construct validity was tested with the method of contrasted groups.18 Construct validity would be confirmed if the median test score of pharmacists was significantly higher than median test score of patients at T1. Mann–Whitney U-test was used to compare numerical data between groups. At T2, patients received educational counseling about DOAC and responsiveness of the KODOA-test was tested. Median test scores before (T2) and after educational counseling (T2 + edu) about DOAC were compared using Wilcoxon test.

The test–retest method was used to demonstrate the stability of the questionnaire over time. Pearson’s correlation coefficient was calculated between the test scores from T1 to T2 and T2 + edu to T3. Additionally, internal consistency or reliability was assessed using Cronbach’s α, which indicates whether each item of a scale is appropriate for assessing the underlying concept of its scale. Values for Cronbach’s α range between 0 and 1; the closer they are to 0, the less the items are related to one another. Values between 0.5 and 0.7 are the minimal requirement to indicate satisfactory internal consistency.19

Index of difficulty and item discrimination (D-value) were calculated to assess performance of single items. The index of difficulty is defined as the proportion of patients answering the item correctly (=number of correct responses/total number of responses). An item with an index of difficulty >0.75 is deemed to be too frequently answered correctly.20 Item discrimination (D-value) tests how well an item discriminates between people who have a low and high knowledge score. A D-value is calculated by subtracting for each item the proportion of respondents answering correctly in the lowest quartile from those answering correctly in the highest quartile, aiming for a mean D-value of 50%.20

Results

Out of 45 compiled items, the expert panel selected 15 to be important for knowledge of DOAC. The 15 items were derived from the educational topics “Application and treatment adherence” (5 items, number [nb.] 3, 5–8), “Risk–benefit of treatment” (2 items, nb. 2, 4), “Accessing health care providers” (2 items, nb. 12, 13), “Self-care” (2 items, nb. 9, 15), “Relevant blood tests” (1 item nb. 14), “Medication interactions” (2 items, nb. 10, 11), and “Mode of action” (1 item, nb. 1). Two versions of the KODOA-test were developed, 1 for patients with AF and 1 for patients with deep vein thrombosis or pulmonary embolism. The 2 versions differed in the answer options in items 2, 3, and 4, depending on the underlying disease (Figure S1).

Out of 67 invited patients, 32 participated (Table 1). They were prescribed either rivaroxaban (84.4%) or apixaban (15.6%). Mean duration of DOAC use was 1.8±1.7 years. Patients had mandatory education (n=3), technical/vocational education (n=11), a higher school certificate (n=11), University degree (n=4), or did not give any information (n=3). A majority of patients stated to have no difficulties to understand written and verbal information about medication and were found to be confident when filling out forms (Table 1). One patient (3.1%) was lost to follow-up (T3) and 1 patient did not fill in the knowledge question for follow-up evaluation.

Table 1

Patient and pharmacists characteristics at T1 (baseline)

Characteristics	Patients (n=32)				Pharmacists (n=28)
Female (%)	65.5				89.0
Age (years; mean ± SD)	73±9				34±9
Indication (%)
AF	62.5
Deep vein thrombosis/pulmonary embolism	37.5
Duration of therapy (years, mean ± SD)	1.8±1.7
Comprehension of written information about medication (%)
Without difficulties	46.8
Neutral	21.9
With difficulties	31.3
Comprehension of verbal information about medication (%)
Without difficulties	75.0
Neutral	12.5
With difficulties	12.5
Confidence when filling out forms (%)
Confidence	93.8
No confidence	6.2
Self-estimated knowledge (% of patients [number of patients])	Bad9.4(n=2)	Moderate31.3(n=10)	Good53.0(n=17)	Excellent6.3(n=2)
KODOA-test scores (mean ± SD)	11.0±4.4	8.2±2.4	10.9±2.1	10.5±3.5
Number of ticks “being certain about the correctness of the given answer” (mean ± SD)	7.7 ±1.5	9.6±3.4	10.9±2.6	12.5±2.1

Abbreviations: AF, atrial fibrillation; KODOA, Knowledge Of Direct Oral Anticoagulants; SD, standard deviation.

Feasibility was confirmed. Mean time to fill in the KODOA-test was 7.8±3.0 minutes, and patients were satisfied with the structure, readability, comprehension, and time to fill in the questionnaire (Figure 1).

Figure 1

Patients’ answers to questions regarding feasibility of the questionnaire (n=32).

Twenty-eight pharmacists (34±9 years, 89% women, 8±9 years working experience) filled in the KODOA-test. Median test score of community pharmacists (13.0 [IQR 1]) was significantly higher compared to test score of patients (10 [IQR 4]) at T1 (U-Test: p<0.001). Median test score of patients at T2 was 11 (IQR 2), and increased significantly to 14 (IQR 3) after educational counseling (Wilcoxon: p<0.001). Educational counseling about DOAC lasted 14±7 minutes. The median number of items whose answers were correct and ticked as confident was calculated. This number increased from 9 (IQR 3) at T2 to 14 (IQR 1) after educational counseling at T2 + edu (Wilcoxon: p<0.001).

Reliability of the KODOA-test was demonstrated. Test–retest reliability between T1 and T2 (r=0.800, p<0.001; 12 day mean duration between the tests) and between T2 + edu and T3 was confirmed (r=0.644, p<0.001; 17 days mean duration between the tests) (Table 2). Internal consistency was acceptable (Cronbach’s α=0.67). Cronbach’s α was found to increase slightly (0.673–0.678) if items 3, 5, 6, or 11 were excluded (Table 3). The index of difficulty for most items was satisfactory (0.38–0.72) (Table 3). The mean D-value was 42.5%.

Table 2

Mean KODOA-test scores, ie, correctly answered questions (± SD) at T1, T2, T2 + edu (after educational counseling), and T3

Visit	Patients (n=32)	Pharmacists (n=28)	p-value
T1	10.03±2.7	13.3±1.0	<0.001
T2a	10.06±2.1*	NA
T2 + edu	13.9±1.2*
T3b	13.1±1.7	NA

Notes:

Mean duration between tests: 12 days.

N=31 for retest score; mean duration between tests: 17 days.

p<0.001.

Abbreviations: KODOA, Knowledge Of Direct Oral Anticoagulants; NA, not applicable; SD, standard deviation; T1, first appointment at the community pharmacy; T2, second appointment at the community pharmacy; T3, 2 weeks after T2.

Table 3

Content and psychometric properties of the KODOA-test items

Item number	Content	Patients with correct answer (%)	Index of difficulty	Item discrimination	Cronbach’s α if item is omitted
1	Name of the DOAC	93.8	0.94	18.2	0.660
2	Duration of therapy	90.6	0.91	18.2	0.669
3	Dosing frequency	81.3	0.81	45.5	0.673
4	Indication for DOAC	93.8	0.94	18.2	0.656
5	What to do if uncertain whether last dose was ingested	43.8	0.44	47.7	0.673
6	What to do in case of a missed dose	37.5	0.38	35.2	0.678
7	What to do in case of double dosing	37.5	0.38	56.8	0.645
8	Does the DOAC work when vomiting immediately after ingestion	37.5	0.38	81.8	0.640
9	Most frequent side effect	59.4	0.59	47.7	0.647
10	Safest OTC analgesic with DOAC	53.1	0.53	47.7	0.640
11	Whom to ask about safe OTC	90.6	0.91	5.7	0.673
12	When to inform others about DOAC therapy	87.5	0.88	27.3	0.655
13	Recognition of emergencies	65.6	0.66	81.8	0.605
14	How often to visit the doctor for lab monitoring	59.4	0.59	51.1	0.652
15	Carrying an anticoagulation card	71.9	0.72	54.5	0.638

Abbreviations: DOAC, direct oral anticoagulants; KODOA, Knowledge Of Direct Oral Anticoagulants; OTC, over the counter.

There was no relationship between KODOA-test scores at T1 and patient self-estimated knowledge of DOAC (Table 1). Patients with higher self-estimated knowledge tended to answer with more confidence (Table 1). At T3, all patients agreed that they knew more about their DOAC after having received educational counseling.

Discussion

The KODOA-test was confirmed to be feasible, comprehensive, reliable, and valid to self-assess patient knowledge of DOAC. Furthermore, the KODOA-test is responsive to change. Content validity was ensured by a developing process with experts of different professions taking care of patients undergoing anticoagulation therapy in their daily practice. Construct validity was supported by significant differences in scores between patients and pharmacists. Finally, the KODOA-test was responsive to educational counseling on DOAC, supporting construct validity.

Patients were able to complete the test within 8 minutes, and rated this time as acceptable. Compared to other assessment questionnaires for knowledge of OAC therapy, the KODOA-test is relatively brief. The AKT and AKA assessment questionnaires need 10–15 and 20 minutes to be filled in, respectively.11,12

The index of difficulty of most items and mean D-value was satisfactory. Although 6 items showed a poor level of difficulty, they were retained because they are related to core information about DOAC therapy, such as name, treatment duration, or whom to ask in case of pharmaceutical questions. The KODOA-test showed moderate internal consistency, with a Cronbach’s α of 0.67. The obtained Cronbach’s α value is likely a result of the multifactorial nature of the KODOA-test because it consists of items derived from several educational topics. Omission of single items had minimal effect on Cronbach’s α. The KODOA-test included very easy items and very difficult items. Despite the fact that such grade of difficulty tends to decrease the internal consistency of the scale,21 HCPs need basic questions on the medication used by the patient such as dosing frequency. Finally, a low Cronbach’s α might be explained by the shortness of KODOA-test with 15 questions, since internal consistency increases as test length increases.22 Therefore, an α of 0.67 seems acceptable.

Stability over time was confirmed by test–retest correlation above the threshold of 0.7, which indicates adequate reliability.23 The mean time between test and retest was 12 days. Recommendations for interval between 2 identical tests vary between 2 days21 and 3 months.24 Two studies investigating knowledge of anticoagulation with questionnaires had longer intervals of 2–3 months for the retest,10,12 and 2 further studies on knowledge of nutrition had intervals of 2 weeks.25,26 We selected a short time interval based on ethical considerations, because patients with deficient knowledge should be corrected as soon as possible in order to avoid life-threatening situations. Knowledge scores remained high after approximately 2 more weeks, demonstrating the sustainability of the educational counseling. In addition, patients agreed uniformly that their knowledge had increased. Nevertheless, further follow-ups could be useful to maintain a high level of knowledge. The state of the art currently in Switzerland and in many other countries is silo interventions from several HCPs. It would be best practice to share information and questionnaire results between HCPs.

The majority of patients rated their knowledge of DOAC at baseline as good or excellent. However, this self-estimation did not correspond to KODOA-test scores, suggesting that patients both overestimate and underestimate their knowledge. Consequently, self-estimation of knowledge with a single question should be taken with caution. Further, asking for confidence when answering the item is subject to the same under- and overestimation. However, because of the low number of patients in the bad and excellent self-estimated knowledge groups, this finding needs further investigation.

Interestingly, patients’ scores after educational counseling were still below the maximum, with an average of 1.9 erroneous answers. The items concerning safety issues (missed dose, double dosing, vomiting) seem difficult to remember. HCPs should be aware that some pharmacological concepts are more demanding than others.

Previous studies observed an increase of knowledge of OAC therapy following education.27–30 However, only 1 study included patients on DOAC, but knowledge assessment methods were not developed for patients on DOAC therapy.27 In a recent study, the development of an OAC knowledge questionnaire was reported.12 The AKT contains 20 items applicable for DOAC patients and 8 additional items for VKA patients. Overall, the AKT seems to be a valid and reliable questionnaire to measure patient knowledge of oral anticoagulants. However, it is not reported whether the AKT is responsive to change, and scoring seems more complicated as with the KODOA-test. In light of these inconveniences, KODOA-test is likely to be easier to use in patient studies.

Our study has several strengths. First, we followed an evidence-based approach to select items. We coupled a literature search to educational theory to reduce the number of items to the minimum needed to assess knowledge. Second, we used the most recent publication such as the EHRA guidelines to cover exhaustively the characteristics of anti-coagulation therapy. Third, we selected experts in all fields of the health care professions to determine the relevant items needed in a self-assessment questionnaire. Fourth, we tested pharmacists as representatives of HCPs. We are confident that similar knowledge results would have been obtained with doctors and nurses. Finally, we developed a manual with answers to obtain standardization of educational counseling. By doing so, we offer a fast and efficient way of counseling to all HCPs who provide information for patients.

We acknowledge some limitations. First, 52.3% of the patients refused to participate in the study. Time consumption (ie, 2 visits at the community pharmacy) was likely the reason of this moderate acceptance rate, rather than a low acceptability to answer the KODOA-test. Second, the relatively small number of subjects may limit the ability to generalize results to all patients on DOAC. Importantly, sample size was adequate to show that the KODOA-test is responsive to change and that construct validity is given (data not shown). Third, this study included elderly Swiss German speaking patients with an indication to AF or deep vein thrombosis/pulmonary embolism and taking rivaroxaban or apixaban, that is, only 2 of the 4 commercially available DOACs in Switzerland. Because the KODOA-test targets DOAC knowledge in general, that is, independently of the anticoagulant agent, we cannot think that patients on other anticoagulants (ie, edoxaban or dabigatran) or in another setting (ie, after surgery) would have answered differently. Applicability of the KODOA-test in other settings should nevertheless be investigated in a further study. Finally, the multiple-choice format permits guessing, which may have increased scores in both the patient and pharmacist groups.

Conclusion

The KODOA-test showed to be valid and reliable in Swiss German speaking elderly outpatients taking DOAC. The application of the KODOA-test in other populations needs confirmation by further research. To our knowledge, the KODOA-test is the first validated questionnaire specific for patients taking DOAC that is responsive to educational counseling. Therefore, the KODOA-test could be used in clinical trials where associations between knowledge of DOAC and adherence or clinical outcomes are of interest.

Questions of the KODOA-test translated to English (for informative purpose; the English version has not been validated).