Evaluation of a Dementia Education Program for Family Medicine Residents.

BACKGROUND: Dementia diagnosis and management is increasing in importance in the training of future family physicians. This study evaluated the impact of a dementia education program for family medicine residents (FMR) on residents' knowledge, attitudes, and confidence with respect to dementia assessment and management. A three-part questionnaire was developed and validated for these purposes.
METHODS: A mixed methods study design was employed. The questionnaire's internal consistency and test-retest reliability was determined and content validity was assessed. Twelve FMR participated in questionnaire validation. Program participants completed the validated questionnaire at baseline, at interim, and following program completion. Twenty-seven FMR completed the questionnaire as part of the program evaluation. Willing residents also participated in program feedback interviews. Differences in questionnaire scores between program participants and the comparison group were examined.
RESULTS: Each questionnaire component demonstrated high internal consistency (Cronbach's α: 0.83-0.91) and test-retest reliability (intraclass correlation coefficients: 0.74-0.91). Program participants (n = 15) scored significantly higher than the comparison group (n = 12) on the knowledge component and also reported greater confidence in several areas. Qualitative data indicated that residents felt the program focused on important topic areas and appreciated the opportunity to work in an interprofessional team.
CONCLUSION: Evaluation results indicate that the program improved FMRs' knowledge on dementia assessment and management, as well as increased the residents' confidence levels.

Introduction

An estimated 480 600 Canadians live with Alzheimer disease or a related dementia.( As the Canadian population ages, it is expected that by 2038, the prevalence of dementia will rise to 1.125 million people.( It is clear that dementia is an issue which affects many Canadians and will continue to affect an increasing number in the near future.

The Canadian Consensus Conference on Dementia has recommended that family physicians play a prominent role in dementia care.( This is fitting in that the majority of older adults receive their care from family physicians.( However, family physicians have been found to lack knowledge and confidence with regard to dementia diagnosis and management.( Family physicians may express significant diagnostic uncertainty( and, consequently, a desire to learn more about dementia.( Surveys of family physicians have also found that they are not satisfied with the dementia training received in medical school and residency programs( and that a need exists for increased coverage of dementia in residency training programs.( Several studies have examined geriatric education programs for medical students and residents; however, few have used validated measures to assess changes in knowledge and confidence. Moreover, as these programs were generally geriatrics-based rather than specifically dementia-focused, they do not make clear the effects of the education programs on participants’ dementia-related knowledge and confidence.

The objective of this research was to evaluate the effectiveness of a dementia education program for family medicine residents (FMR) on residents’ knowledge, attitudes, and confidence with respect to dementia assessment and management through a validated online questionnaire and post-intervention interviews.

METHODS

Questionnaire Development and Validation

The evaluation questionnaire was developed in collaboration with the physician investigator (LL). The questionnaire consisted of a 12-item multiple choice knowledge component, an 8-item component assessing attitudes toward working with various age groups, and a 26-item component assessing attitudes and confidence toward the diagnosis and management of dementia. To develop the knowledge component, topic areas related to dementia that were deemed essential knowledge for graduating FMRs were identified from the McMaster University Department of Family Medicine Postgraduate Residency Care of the Elderly Objectives.( In order to develop the attitudinal, comfort, and confidence components, current literature was examined.( The majority of existing questionnaires, surveys, and scales identified were largely geared toward family physicians. Pertinent questions were identified by one author (JCP) and modified to ensure they were applicable to FMR. Additionally, LL provided input regarding modifications and the suitability of questions, and developed additional questions relevant to the specific content of the education program. The attitudinal, comfort, and confidence items were structured as statements to which residents were asked to provide their levels of agreement on seven-point Likert scales.

The questionnaire was tested among 12 FMR not participating in the program. Test–retest reliability was assessed and a content validity matrix was constructed. A content validity matrix consists of topic areas on one axis and questionnaire items on the other axis. Each item should fit within at least one topic area and all topic areas should include at least one item. The content validity matrix was completed to ensure that the questionnaire provided adequate coverage of the areas of interest. Participants were also asked to complete cognitive interviews( which were recorded, transcribed verbatim, and analyzed for common themes. Cognitive interviews require the participants to vocalize their thought processes and interpretations as they complete the questionnaire. The questionnaire was revised, based on the results of these assessments.

Dementia Education Program Participant Recruitment

Postgraduate year 1 and 2 FMR from the Michael G. DeGroote School of Medicine (McMaster University) Kitchener satellite campus (intervention group) and Hamilton site campus (control group) were recruited for the evaluation. Nineteen FMR in Kitchener were eligible to participate in the program.

Intervention

The dementia education program consisted of a lecture component, a half-day interactive workshop, and a clinical practice component. The lecture component was an academic half-day (AHD), which included the following topics: clinical reasoning approach and office-based assessment tools for suspected cognitive impairment; differentiating normal aging, mild cognitive impairment (MCI), and dementia; distinguishing among the types of dementias; a rational approach to investigation and treatment of dementia; and management of driving concerns.

The classroom-based component of the program was a half-day interactive workshop, which allowed residents to apply what they had learned in the AHD to anonymous cases from an on-site memory clinic. This included interpreting completed assessment tools, arriving at diagnoses, and developing management plans. The workshop was facilitated by the physician investigator (LL), together with two nurses from the memory clinic and a clinical pharmacist.

The final component allowed residents to apply skills they had learned in the previous two sessions in a clinical setting. Working in an interprofessional team with a social work intern and school of pharmacy co-op student, each resident had the opportunity to assess two patients in a one-day session at a primary care physician-supervised memory clinic. Residents completed the memory clinic day as early as two days after the case-based workshop and as late as four months after the workshop, due to scheduling complications. This was an unavoidable limitation of the program design.

Consent to participate in the evaluation of the training program was obtained from all participants prior to the start of the program. Residents were asked to complete an online version of the questionnaire prior to the start of the program to provide a baseline assessment following the half-day case-based workshop to provide an interim assessment, and after completion of their memory clinic day to provide a final assessment. In addition to being prompted to complete the questionnaire following the memory clinic, residents were also invited to participate in an individual interview. The aim of the interview was to expand on topics covered in the questionnaire and to solicit feedback on the perceived relevance and usefulness of the training program

Statistical Analyses

This study was approved by the University of Waterloo’s Office of Research Ethics. Descriptive statistics from the questionnaire were analyzed and reported. A pre/post-test design was originally planned; however, the limited availability of the residents to complete the data resulted in a very low yield of complete pre/post data (20%) and the planned analyses were subsequently modified. It was ascertained that the scores from FMR who participated in the test–retest of the questionnaire could serve as a comparison group, as these residents did not participate in the dementia education program or have any involvement in a memory clinic. Given that these residents completed the questionnaire twice, data from the first completion were used, as this would minimize potential testing effects. Demographic and other descriptive characteristics were compared to determine that the intervention and control groups were similar.

The knowledge component and attitude and confidence components of the questionnaire were each scored. Mean scores on a seven-point Likert scale were calculated for each questionnaire item in the latter two components. Questionnaire items in the knowledge component were each assigned a point value depending on the number of correct responses to the questionnaire items. If an incorrect response was checked, a point would be deducted. If a response was missed, no point was awarded. A mean score was calculated for each group on each knowledge question and a mean difference was calculated.

Qualitative Analyses

Five FMR were willing to participate in interviews. These five individuals did not differ with regard to demographics from FMR who did not participate in interviews. The five interviews were recorded and transcribed verbatim. Transcripts were analyzed consistent with methods recommended by Creswell.( The transcripts were first coded into emergent categories and these were then organized into broader themes. All coding and organizing of themes was conducted by one member of the research team (JCP). An attempt was made to interconnect themes, where possible, in order to create an inclusive description of the data. The themes were interpreted with the aim of identifying effects on residents’ knowledge, attitudes, and confidence, as well as successful and unsuccessful aspects of the dementia training program and recommendations to improve its future operation. A concurrent triangulation strategy was employed, with the findings from the quantitative and qualitative analyses compared at the interpretation stage.( The aim of this comparison was to identify the results that were supported by both analyses, as well as those that were not, and to help determine reasons for observed dissimilarities. In this manner, the quantitative and qualitative results informed the final conclusions of this research.

RESULTS

Reliability and Content Validity of the Questionnaire

Twelve FMR from the Hamilton, Ontario site participated in assessing the reliability and content validity of the questionnaire. Table 1 presents the intraclass correlation coefficients and Cronbach’s α statistics for each of the three components of the questionnaire. Appendix 1 outlines the questionnaire items which fulfilled each domain of the content validity matrix. Kappa statistics for each questionnaire items and copies of the questionnaire are available from the first author upon request. Two items were added in order to address two topic areas of the content validity matrix which were not covered by any questionnaire items. Analysis of cognitive interviews resulted in modification of the wording of one of the attitudinal questionnaire items, as the wording was found to be too extreme.

TABLE 1.

Test-retest reliability and internal consistency for each questionnaire component

	Test-Retest Reliability: Intraclass Correlation Coefficient (95% CI)	Internal Consistency: Cronbach’s α
Knowledge Component	0.89 (0.75–0.96)	0.83
Attitudes Towards Working with Various Age Groups	0.91 (0.79–0.97)	0.91
Attitudinal and Comfort/Confidence Component	0.74 (0.32–0.92)	0.90

Questionnaire Results

Analyses were conducted comparing scores from the 15 Kitchener site FMR (intervention) with the scores of the 12 Hamilton (control) site residents (Table 2). With regard to demographics, the groups were found to be similar with the exception of age (p = .02). Significant differences in score were found between individual knowledge items as well as total knowledge score (Table 3).

TABLE 2.

Characteristics of the Kitchener and Hamilton site residents

		Kitchener Residents (Intervention, n=15)	Hamilton Residents (Control, n=12)
Mean age in years (Standard error)		27.8 (0.52)	31.8 (1.50)
Gender distribution (Number of individuals)		Female: 8Male: 7	Female: 7Male: 5
Residents with previous undergraduate MD training in geriatrics (%)		10 (66.7%)	4 (33.3)
Residents with previous undergraduate MD training in dementia (%)		9 (60.0)	3 (25.0)
Residents with personal experiences with dementia (%)		7(46.7)	7 (58.3)
Participation in components of dementia education program	Academic Half-Day (%)	13 (86.7)	0 (0.0)
	Case-based Workshop (%)	11 (73.3)	0 (0.0)
	Memory Clinic (%)	14 (93.3)	0 (0.0)

TABLE 3.

Knowledge-based questionnaire items demonstrating significant differences in scores between Kitchener and Hamilton site residents

Knowledge-Based Questionnaire Item	Mean Difference[a] (95% CI)	Significance value
2. The following medications should be avoided if possible in cognitively impaired patients (check all that apply): lorazepam ASA dimenhydrinate amitriptyline ditropan	0.93 (0.06–1.80)	0.037
3. The best way of differentiating mild cognitive impairment from dementia is by (check one): CT scan MRI functional abilities assessment MMSE assessment of ability to understand proverbs	0.44 (0.07–0.81)	0.022
6. In a patient with cognitive impairment, the following should be considerations in assessing fitness to drive (check all that apply): visuospatial function executive function Trails B test degree of cognitive impairment ability to pass Ministry of Transport drivers assessment required every 2 years after age 80	1.68 (1.04–2.32)	<0.001
Total Knowledge Score	4.54 (2.38–6.69)	<0.001

All differences in score are in the positive direction.

No significant differences were found on the second component of the questionnaire which measured participants’ preferences working with various age groups. However, several significant differences were identified between participants’ scores on the attitudinal and comfort/confidence component (Table 4).

TABLE 4.

Questionnaire items demonstrating significant differences in scores between Kitchener and Hamilton site residents

Questionnaire Item	Mean Difference [a] (95% CI)	Significance value
2) h) I feel confident managing dementia in older patients.	1.08 (0.15–2.01)	0.025
2) k) I can differentiate between different types of dementia.	1.30 (0.10–2.49)	0.035
2) q) I am confident in my ability to prescribe appropriate pharmacotherapy, if necessary, when managing patients with dementia.	1.17 (0.23–2.11)	0.017
2) r) I am likely to refer patients with mild cognitive impairment to a Memory Clinic.	1.27 (0.26–2.29)	0.016
2) t) I am likely to refer patients with dementia to a Memory Clinic.	1.27 (0.50–2.05	0.003

All differences in score are in the positive direction.

Feedback Interview Results

Five residents participated in feedback interviews on the dementia education program. Five common themes were identified. Illustrative quotations are presented in support of the themes.

Sequence and Organization of Program Was Conducive to Learning

All five interview participants commented on finding the organization of the program, as well as the sequence of its components, to be conducive to learning about dementia diagnosis and management.

I think they all complemented each other so well, that it was, each component was a natural progression into the next and you know one component solidified a previous component very well.

Interprofessional Experience Was Greatly Valued

The dementia education program was designed to include an interdisciplinary aspect which was found to be greatly valued by the participants. One participant provided an example of how she found it beneficial to work with one of the allied health professionals:

She (patient) was having problems with her stomach and she thought it was related to the medication that she was taking. So then the pharmacy student was able to jump in right away and able to give their take. And you know, we know a lot about medications but we don’t necessarily know all of the side effects and interactions, so that was a really great experience and I learned from them and I think they also learned from me. And from what I’ve seen that’s how medicine is supposed to be practiced and it’s often not practiced like that anywhere, it’s just too much rush-rush in the clinic. And I just when I left there that day I thought this is what I went into medicine for, to have an experience like that.

Dementia Assessment and ManagementIs an Undertrained Area

All five of the participants commented that they felt that dementia assessment and management was an area that was not sufficiently covered both in undergraduate medical education and residency training.

I think that that’s a topic that’s poorly taught in medical school and not a topic that I thought I had a handle on at all.

Improved Knowledge

Each of the participants expressed that they felt that their knowledge regarding dementia assessment and management had improved as a result of the program. Although various areas of improved knowledge were mentioned, the topic areas of pharmacotherapy and driving-related issues were particularly brought up as ones in which the participants felt that they both struggled the most and learned the most.

I don’t even think before I knew what MCI (mild cognitive impairment) was. I mean I knew what dementia was and I knew just basic important things to remember about each type so that I could pass the licensing exam. But I don’t think anyone’s ever talked to me about the concept of MCI.

One of the most practical things I learned about was driving, which I really had no idea about beforehand.

I think maybe the most valuable thing actually is the things to avoid. All those medications to look for that could be contributing to a dementia.

Minor Effect on Interest in Managing Older Patients

Most of the participants commented that the program had a minor effect on their interest in managing older patients. In some cases this was due to an already high interest in older populations, while in others it was that the program had not affected their interest significantly. One participant expressed in detail her frustrations working with older patients as a result of the structure of primary care and how this has affected her interest level.

If I could deal with older patients in a setting like this I think it would be absolutely wonderful for physicians and patients. ...You know I really enjoy working with older people but it’s just the way things are set up right now that you know, I only book 10 minutes and these people have multiple problems and I don’t feel like I’m doing a good job if I don’t deal with them. So I do deal with all their problems and then your office is running 45 minutes behind which is not really fair to your other patients. So that’s a big problem I find with dealing with the older patients and that’s my only I guess reason why I kind of avoid it.

DISCUSSION

This study highlights the need for dementia training in post-graduate family medicine programs. Data obtained from the questionnaire and the feedback interviews appear to be convergent. The questionnaire results are consistent with the qualitative findings which can suggest some evidence of an effect of the program. For example, the difference in total score on the knowledge component of the questionnaire was significantly improved. Comments by participants in the follow-up interviews indicate that the participants’ also strongly felt that knowledge was improved.

In addition to total knowledge score, which demonstrated a large and significant difference between groups, scores on three knowledge-based items were also significantly higher in the Kitchener site group. One of these items was related to driving, while another to medications. Interestingly, participants in the feedback interviews specifically mentioned driving-related issues and pharmacotherapy as two key areas where they felt their knowledge had improved. Related to the comfort and confidence component of the questionnaire, participants demonstrated statistically significant improvements in their reported confidence managing dementia and prescribing appropriate pharmacotherapy. These findings are once again consistent with what was expressed by participants in the feedback interviews. Moreover, pharmacotherapy has been identified in the literature as a particularly challenging area for practicing family physicians.(

A significant difference also existed with regard to referrals to memory clinics, with participants from the Kitchener site being more likely to do so. This indicates that it is likely that exposure to a memory clinic through the program made the participants more aware of memory clinics, how they can be useful, and how family physicians can benefit from such clinics. Combined with their increased knowledge, it may be likely that participants from the Kitchener site may be more likely to make appropriate referrals.

Much in line with the shift to interprofessional care teams in health care,( the interprofessional aspect of the program was also found to be valuable by the residents. When asked what they could take away from the program, participants most often responded by describing the interprofessional experience, which they found to be applicable to other areas of medicine outside of dementia and geriatrics. One participant provided a detailed example, concluding with the strong statement that these were the types of experiences for which she had gone into medicine. Her statement seemed to typify the positive feelings expressed by most intervention participants.

The feedback interviews also indicated that dementia is often an undertrained area for undergraduate medical students and residents. The admission by one of the residents that she had simply memorized what was needed to pass the licensing exam was very telling. It further emphasizes the need for a program such as the dementia education program, and underscores the need for additional attention to dementia and geriatric medical education which has been identified in the literature.

Interestingly, the program appeared to have little effect on participants’ interests in working with the elderly and individuals with dementia. This was demonstrated both by the questionnaire, on which no items related to interest demonstrated significant differences, as well as by comments made by participants in the feedback interview. No dissimilarities were identified between the questionnaire scores and themes generated from the qualitative data when employing the concurrent triangulation strategy. Similar results have been identified in the research literature with other health-care professionals, such as nurses. Generating interest in caring for this specialized population has also been a challenge in these professions.(

This study was limited by low pre-test/post-test response rates (20%), which were not sufficient for powered analyses. Moreover, the low sample sizes amongst the two groups used for comparisons may have resulted in the inability to detect differences between the groups, both with regard to demographics and changes as a result of the program. It should also be noted that possible differences in training between the two sites, aside from the program, were not accounted for in analyses and that the generalizability of the results may be limited, as the study was conducted at one medical school. The low sample size also resulted in point estimates with wide confidence intervals, particularly for the reliability calculation of the attitudinal/confidence component. A larger sample size would result in point estimates with narrower confidence intervals, which could provide a better indication of the reliability of the questionnaire items.

The results of the comparisons taken together with the qualitative data through a concurrent triangulation strategy indicated several benefits of the program identified by its participants. While dissemination of this program may bring logistical challenges, such as developing memory clinics in locations where they do not currently exist, the two components of the program that did not require clinical work were still deemed by participants to be very useful on their own. These components are relatively simple to implement and at the minimum should be considered in cases where memory clinics are not available. Following the results of this evaluation, there may be increased support for this program more broadly. The results of this study underscore the need for increased dementia training in family medicine residency programs. Given the future projections of dementia prevalence, this need will only become more pronounced if additional training is not implemented. Further work in this area is encouraged to identify successful training programs and modes of delivery. It is hoped that by improving the training provided to FMR, the care received by patients will be improved as well.

Appendix 1:

Content Validity Matrix Domains and Questionnaire Items Addressing each Domain

Domain	Questionnaire Item
Appropriate use of medications (knowledge)	Knowledge component: Items 1, 2, 4
Appropriate use of medications (confidence)	Attitudinal component: Item q)
Differentiating between normal ageing, MCI, and dementia (knowledge)	Knowledge component: Item 3
Differentiating between normal ageing, MCI, and dementia (confidence)	Attitudinal component: Item k)
Appropriate use and interpretation of common cognitive tests (knowledge)	Knowledge component: Items 3, 9
Appropriate use and interpretation of common cognitive tests (confidence)	Attitudinal component: Items l), m), n), o)
Appropriate management of driving issues with patients with dementia (knowledge)	Knowledge component: Item 6
Appropriate management of driving issues with patients with dementia (confidence)	Attitudinal component: Items x), y)
Attitudes toward working with allied health professionals	Attitudinal component: Items w)
Referral practices for patients with MCI or dementia	Attitudinal component: Items r), s), t), u)
Level of comfort working with older adults	Attitudinal component: Item b)
Level of interest in working with older adults	Preference working with various age groups component Attitudinal component: Item a)
Attitudes toward working with older adults	Attitudinal component: Item f)
Comfort communicating with patients/families with cognitive impairment	Attitudinal component: Items c), z)
Comfort communicating diagnosis to patients with MCI/dementia	Attitudinal component: Item d)
Comfort managing comorbidities in older adults	Attitudinal component: Items e), i), j)
Ability to differentiate between the most common types of dementia	Knowledge component: Items 5, 10, 11
Appropriate investigations in patients with cognitive impairment	Knowledge component: Item 3 Attitudinal component: Item p)
Desire to manage patients with cognitive impairment	Attitudinal component: Item i)
Management of patients with dementia (knowledge)	Knowledge component: Items 8, 12
Management of patients with dementia (confidence)	Attitudinal component: Items h), v)