Canadian Indigenous Cognitive Assessment (CICA): Inter-rater reliability and criterion validity in Anishinaabe communities on Manitoulin Island, Canada.

INTRODUCTION: Despite increasing dementia rates, few culturally informed cognitive assessment tools exist for Indigenous populations. The Canadian Indigenous Cognitive Assessment (CICA) was adapted with First Nations on Manitoulin Island, Canada, and provides a brief, multi-domain cognitive assessment in English and Anishinaabemowin.
METHODS: Using community-based participatory research (CBPR) methods, we assessed the CICA for inter-rater and test-retest reliability in 15 individuals. We subsequently evaluated validity and established meaningful CICA cut-off scores in 55 individuals assessed by a geriatrician.
RESULTS: The CICA demonstrated strong reliability (intra-class coefficient = 0.95 [0.85,0.98]). The area under the curve (AUC) was 0.98 (0.94, 1.00), and the ideal cut-point to identify likely cases of dementia was a score of less than or equal to 34 with sensitivity of 100% and specificity of 85%. DISCUSSION: When used with older First Nations men and women living in First Nations communities, the CICA offers a culturally safe, reliable, and valid assessment to support dementia case-finding.

INTRODUCTION

Globally, the proportion of the population over 65 years of age is rapidly growing and the prevalence of dementia is likewise expected to rapidly expand over the next 20 years. , As our collective understanding of the natural course of dementia has progressed, so too has our understanding of the critical importance of early detection and diagnosis to guide treatment and family care plans. , Likewise, there is growing recognition that existing cognitive assessments are not always suitable for diverse subgroups within populations. Many of the existing cognitive assessment tools that are widely available were developed in homogenous, predominantly English‐speaking groups in ways that have not sufficiently addressed cross‐cultural test fairness.

The need for cultural fairness in dementia assessment is urgent, given that the incidence and prevalence of dementia are not consistent across subgroups. Past research has shown that Indigenous people have the highest dementia risk, or equal risk to African Americans, while Asian people show the lowest dementia risk. , , From 2000 to 2012, the prevalence of dementia appears to have declined for White and Black adults, but the opposite was true for Indigenous people. Dementia prevalence in First Nations people in Alberta, Canada, is higher than that in non‐First Nations populations and is increasing more quickly.

Despite the importance of early diagnosis of dementia within Indigenous communities, care providers have noted important limitations in existing methods of assessment. , Qualitative work with Indigenous communities characterized existing tools as culturally inappropriate. The lack of appropriate tests has led to care providers improvising or adapting existing methods without verification of the reliability and validity of their modifications. Providers may use popular clinical tests such as the Mini‐Mental State Examination (MMSE) as a “template” around which they use conversation and select portions, but not fully scored administrations, of the test.

RESEARCH CONTEXT

Systematic review: The authors responded to Indigenous community priorities and reviewed the published literature. While there is limited research on dementia in Indigenous populations and relevant cognitive assessment tools, there are several key publications on these topics which have been cited in this article.

Interpretation: This Indigenous community‐based research resulted in the development and validation of a cognitive assessment tool for use with Indigenous people that integrates culturally safe, trauma‐informed approaches. The study contributes to international work on the importance of culturally adapted approaches to cognitive assessment.

Future directions: The article describes the initial findings in developing evidence for validity for a culturally grounded Indigenous cognitive assessment tool. Future directions of this work include testing of the Canadian Indigenous Cognitive Assessment (CICA) in diverse Indigenous populations, assessing the ability of the tool to detect cognitive decline, and implementation of the cognitive assessment tool in First Nations communities.

Given the evidence of a need for culturally appropriate dementia screening tools for Indigenous people in North America, we report initial results from test–retest reliability and criterion validity for a newly adapted instrument. The Canadian Indigenous Cognitive Assessment (CICA) was adapted from the Kimberley Indigenous Cognitive Assessment (KICA), a culturally safe tool developed with Indigenous older adults in the Kimberley region of Australia. The CICA was adapted using extensive community‐based participatory research (CBPR) in collaboration with First Nations communities on Manitoulin Island, Ontario, Canada. The tool is available in English (Appendix A in supporting information) and Anishinaabemowin (Appendix B in supporting information). Using a CBPR approach, we aimed to assess the reliability of the CICA assessment and scoring procedures across administrations by two different evaluators. We also aimed to assess the validity of the CICA, as measured by the sensitivity and specificity of the CICA in detecting cognitive impairment identified in a clinical examination by a physician.

METHODS

Community context and engagement

This reliability and validity study was an extension of the community‐based dementia research that the First Nations communities on Manitoulin Island, Ontario, Canada, and academic partners have been working on over several years. , First Nations are one of three culturally and politically distinct Indigenous groups within Canada. The six communities on Manitoulin Island who collaborated in this study are Anishinaabe communities. On Manitoulin Island, 41% of the population is Indigenous and ≈5% of the adult population speaks Anishinaabemowin at home. The research was conducted using Indigenous research methodologies in combination with CBPR approaches. Indigenous research methodologies prioritize Indigenous knowledge and expertise and hold the teachings of knowledge keepers as equal to Western medicine and academia. Further, there is a recognition of the continued effects of colonization on Indigenous peoples and steps are taken to conduct research “with,” not “for” or “about,” communities. The research was guided by a Community Advisory Council (CAC) composed of 16 Anishinaabe adults, caregivers, health‐care providers, and Elders from participating communities who meet regularly to discuss and direct projects related to aging and dementia. The research team included a Community Researcher with more than two decades of community nursing experience in the region who is fluent in Anishinaabemowin, the first language of many older Anishinaabe adults in the area.

Ethical approval

The protocol was reviewed and approved by the Laurentian University Research Ethics Board, the Manitoulin Anishinaabek Research Review Committee, and the CAC. In addition, Band Council and health authority resolutions or motions, signifying the approval from First Nations governance to conduct the research, were sought as appropriate.

Reliability testing

We conducted the reliability testing in July 2017. In alignment with sampling methods used for the reliability testing for the KICA in Australia, we recruited 15 participants using purposive and snowball sampling. We aimed for an equal number of men and women and participants with a range of suspected impairment levels (none, mild impairment, dementia). The sample size is adequate to support 90% agreement between raters, with a 30% error rate. Local health‐care providers and CAC members connected potential participants and proxy decision‐makers, as needed, to the Community Researcher. We included participants as young as age 45 years, due to recent findings indicating the earlier onset of dementia in Indigenous populations in Canada, and no upper age limit was applied.

The CICA was completed and scored with each participant by one of two assessors in the morning, and subsequently by the other assessor, blinded to the initial results, a minimum of 5 hours later. As such, both inter‐rater reliability and test‐retest reliability were being evaluated simultaneously. An experienced and trained translator was used for those participants who preferred to speak in Anishinaabemowin. Assessments occurred in a comfortable location that was determined according to the participant's preference, including participants’ homes, a long‐term care facility, or a local health center. An honorarium of $25 CDN was provided to participants. Each completed CICA was scored using standardized scoring instructions embedded in the tool and described in the accompanying Guidebook and Instruction booklet (www.i‐caare.ca; Appendices C and D in supporting information).

Reliability was assessed by calculating agreement between the scores produced by the two independent assessors using the intra‐class coefficient (ICC). We also calculated agreement for each of the 11 domains of cognition assessed. Analyses for the ICC and the 95% confidence intervals were completed using SPSS 24 with a two‐way mixed‐effects model intended to be used for a single rater with absolute agreement.

Validation testing

Validation testing began in April 2018 with a new set of participants recruited using purposive and snowball sampling. We aimed to recruit 70 individuals, in alignment with the methods used to validate the KICA in Australia. To ensure that we included people with a range of cognitive impairments, we aimed to include one third of participants who had no suspected cognitive impairment, one third who had some level of suspected impairment, and one third who had known dementia. Every effort was made to recruit equal numbers of men and women and a lower age limit of 45 years was applied. Local health‐care providers and CAC members connected potential participants and proxy decision‐makers, if needed, to the Community Researcher.

A total of 95 individuals discussed participation with research staff; four declined to participate and two lost contact over the course of the study. Once the recruitment goal of 70 participants was reached, research staff canceled the assessments for 19 potential participants due to over‐recruitment. In total, 70 Anishinaabe adults consented to participate and presented for the CICA assessment with the research team. However, only 55 participants fully completed both the CICA and the clinical assessment. One participant was not able to reply to chapters 7 through 10 of the CICA due to vision loss. We prorated the score for the individual but excluded the data from the analysis. Fifteen participants did not present for the clinical assessment due to weather conditions, the death of a prominent community member, community events conflicting with their appointment, or other barriers.

Assessments were conducted in a location convenient and comfortable to participants. The research team coordinated with local medical transportation to assist with travel, as needed, and translation needs were considered at every level of assessment. An honorarium of $50 CDN was provided to participants for their contribution to the validity testing. The CICA was conducted by a trained member of the research team following the standardized administration guidelines. Each completed CICA was scored using standardized scoring instructions embedded in the tool and described in the accompanying Guidebook and Instruction booklet (www.i‐caare.ca; Appendices C and D). Within a subsequent 2‐week period, participants underwent a clinical assessment with one of two geriatricians scheduled over 6 days. One geriatrician has extensive experience working directly with Anishinaabeg older adults in clinical contexts on Manitoulin Island. The geriatricians were blinded to the results of the CICA and provided a professional opinion of probably “normal cognition,” “mild cognitive impairment (MCI),” or “dementia.” The clinical assessment consisted of a semi‐standardized clinical interview with the participant and a family member or caregiver and a physical examination. The geriatrician was free to use clinical judgment in applying existing cognitive tools, including the MMSE, Montreal Cognitive Assessment (MoCA), Geriatric Depression Scale (GDS), and Rowland Universal Dementia Assessment Scale (RUDAS). The KICA tool was not used in the geriatrician's clinical assessment. The geriatricians reviewed each other's assessments and, in cases of disagreement, discussed cases and came to a consensus on a probable diagnosis. Each participant's clinical assessment results were forwarded to the participant's primary health‐care provider, as indicated by the participant or their proxy decision‐maker in their consent form.

Using the clinical assessment as a “gold standard,” we examined a range of potentially appropriate cutoffs for the CICA using receiver operator characteristic (ROC) curve analysis, overall, by the language used during the CICA assessment, and for males and females separately. To do this, sensitivity (true‐positive probability) was plotted against 1‐specificity (false‐positive probability) and examined for relative utility based on the following guidelines: 0.5 to 0.7 = poor, 0.7 to 0.9 = useful, 0.9+ = accurate. Due to the importance of early diagnosis and early intervention in dementia, we emphasized higher sensitivity in our assessment of the cut‐off points for probable dementia and MCI. A minimum alpha level of 5% was set for all analyses (P < .05).

We also explored the relative ability of each cognitive domain that is assessed using the CICA to differentiate between a geriatrician assessment of dementia and no dementia using stepwise discriminant function analysis (DFA; using a significance threshold of 0.15 to enter and stay in the model using Wilks’ method) and forward stepwise logistic regression (using 0.30 significance threshold to enter the model and 0.35 to stay using Wald's method). All analyses were undertaken using SPSS 24 by members of the research team who had no knowledge of the participants.

RESULTS

Reliability

Of the 15 participants in the reliability sample, seven were female and eight were male. The age range was from 45 to 96 years. Six participants were living with pre‐established cognitive deficits due to age‐related dementias; the remaining participants were either cognitively well or unsure if cognitive impairment was a concern. The sample included participants at a range of cognitive levels, with resulting CICA scores from 22 to 39 out of a possible 39 points.

The ICC between the total CICA scores generated by two independent assessors on the same day was 0.94 (0.85, 0.98), which indicates excellent reliability. On average, the scores were 0.6 points higher in the afternoon assessment (P = .23). While this was not a statistically significant difference, it may indicate some degree of learning or comfort that affected the scores slightly when the test was administered twice in the same day. In addition, the difference in scores fell within an acceptable range of ± 2 standard deviations of the mean difference, with the exception of one outlier.

Validity

Of the 95 individuals initially approached to participate in the study, 55 participants were assessed both with the CICA and by the geriatrician. Of the 55 full assessments, 35 were conducted in English, 15 in Anishinaabemowin, and 5 in English to Anishinaabemowin using a translator. For the analysis, we combined the participants who were tested in Anishinaabemowin directly and using a translator to avoid an extremely small sample size for the ROC analysis. Of these 55 who had both assessments, 25 (45.5%) were female. The average age of the participants was 70.2 years (median = 73.0, standard deviation [SD] = 12.1 years, two participants’ age was missing). The clinical assessments indicated that 16 (29.1%) had probable dementia, 19 (34.5%) had probable MCI, and 20 (36.4%) were cognitively normal.

Table 1 shows the results of the ROC analysis, including the sensitivity, specificity, and area under the curve (AUC) for various cut‐points on the CICA score. The best overall performance of the tool for predicting probable dementia was achieved for the cut‐off score of < or equal to 34 out of 39 possible points. At this cut‐point, the AUC was 0.98 (0.94,1.00), sensitivity was 100%, and specificity was 85%. This cut‐point of 34 was valid for both men and women (men: sensitivity of 100% and specificity of 79%, AUC = 0.95 [0.88–1.00]; women: sensitivity of 100% and specificity of 90%, AUC = 1.00 [1.00–1.00]). This cut‐point of 34 was also valid for both English and Anishinaabemowin and English to Anishinaabemowin using a translator (English: sensitivity of 100% and specificity of 87%, AUC = 0.97 [0.92–1.00]; Anishinaabemowin and English to Anishinaabemowin using a translator: sensitivity of 100% and specificity of 78%, AUC = 0.99 [0.96–1.00]). For distinguishing probable MCI plus dementia from normal cognition, the most appropriate cut‐point was < or equal to 37 out of 39 where sensitivity was 97% but specificity was only 40% (AUC = 0.83 [0.73–0.94]). The AUC was 0.83 (0.67–0.99) for females and 0.84 (0.69–0.98) for males. When the analysis for this cut‐point was done by language, the AUC was 0.81 (0.67–0.95) for English speakers and 0.75 (0.55–0.95) for Anishinaabemowin speakers.

TABLE 1

Sensitivity, specificity, and area under the curve for various CICA cut‐off points

Cognition	N	Cut‐off	Sensitivity	Specificity	LR+	LR–	AUC
Dementia vs. (MCI a and normal cognition b )	54	< = 33	0.93	0.97	31	0.07	0.98(0.94,1.00)
		< = 34	1	0.85	6.67	0
(Dementia and MCI) vs. normal	54	< = 34	0.56	0.9	5.6	0.49	0.83(0.72,0.93)
		< = 35	0.65	0.8	3.25	0.44
		< = 36	0.76	0.65	2.17	0.37
		< = 37	0.97	0.4	1.62	0.08
		< = 38	1	0.1	1.11	0
Normal vs. (MCI+dementia)		> = 33	1	0.44	1.78	0	0.83(0.72,0.93)
		> = 34	0.9	0.56	2.04	0.18
		> = 35	0.8	0.65	2.28	0.31
		> = 36	0.65	0.76	2.71	0.46
		> = 37	0.4	0.97	13.33	0.62
MCI vs. normal N = 39	39	< = 34	0.21	0.9	2.1	0.88	0.70(0.53,0.86)
		< = 35	0.37	0.8	1.85	0.79
		< = 36	0.58	0.65	1.66	0.65
		< = 37	0.95	0.4	1.58	0.13
		< = 38	1	0.1	1.11	0
Dementia vs. MCI N = 34	34	< = 33	0.93	0.95	18.6	0.07	0.96(0.89,1.00)
		< = 34	1	0.79	4.76	0
Dementia vs. (MCI+normal) among English speakers N = 34	34	< = 33	1	0.97	33.33	0	0.97 (0.92, 1.00)
		< = 34	1	0.87	7.69	0
Dementia vs. (MCI+normal) among Anishinaabemowin speakers and English to Anishnaabemowin using a translator N = 20	20	< = 33	0.91	1	NaN	0.09	0.99 (0.96, 1.00)
		< = 34	1	0.78	4.54	0
Dementia vs. (MCI+normal) among females N = 25	25	< = 33	1	1	NaN	0	1.00 (1.00, 1.00)
		< = 34	1	0.9	10	0
Dementia vs. (MCI+normal) among males N = 29	29	< = 33	0.9	0.95	18	0.11	0.95 (0.88, 1.00)
		< = 34	1	0.79	4.76	0

Notes: The best overall performance of the tool for predicting probable dementia was achieved for the cut‐off score of less than or equal to 34 out of 39 possible points. At this cut‐point, the sensitivity of the CICA was 100% and specificity was 85%.

Abbreviations: AUC, area under the curve; CICA, Canadian Indigenous Cognitive Assessment; MCI, mild cognitive impairment.

Suspected MCI.

Cognitive impairment was not suspected.

The stepwise DFA revealed very poor performance of the Praxis domain in predicting dementia in this sample. Overall, five domains (Registration, Verbal Comprehension, Visual Naming, Free Recall, and Cued Recall), entered the model (Wilks’ λ = 0.24, F (5, 48) = 48.00, P < .0001). Of these, free recall, registration, and verbal comprehension had higher standardized canonical discriminant function coefficients (0.85, 0.72, − 0.68, respectively), while the remaining two, Visual Naming and Cued Recall, had lower standardized canonical discriminant function coefficients (0.40 and − 0.32, respectively).

A DFA model with just the registration and free recall domains accounted for 63.0% of the variance in a physician's assessment of dementia (Wilks’ λ = 0.37). The standardized canonical discriminant function coefficient for free recall and registration were 0.74 and 0.59, respectively, and 92.6% of the cross‐validated grouped cases were correctly classified. Likewise, a forward stepwise logistic regression with just these two domains yielded Cox and Snell's R2 = 0.54 and Nagelkerke R2 = 0.78.

DISCUSSION

The CICA shows strong evidence of reliability and validity in both men and women and was acceptable to First Nations members and health‐care providers in Anishinaabe communities on Manitoulin Island, Ontario, Canada. While a clear cut‐off score for dementia case‐finding emerged for the CICA, we were less able to detect cases of MCI; however, the results indicate a potential cut‐off, pending further research with diagnostically heterogeneous samples. While some parts of the assessment (Ch. 9, Ch. 3, and Ch. 4), are more important for identifying the most common dementia cases, our findings indicate that it is best to keep the whole assessment intact. Though the praxis domain performed most poorly, praxis impairment can be an important early indicator of atypical dementias such as dementia due to corticobasal degeneration, which we may not have seen in our small sample.

The current study found the CICA had a sensitivity of 100% and a specificity of 85% at identifying dementia (cut‐point of 34/39), which is similar to the classification accuracy for identifying dementia reported for other cognitive screening tools. A recent review of studies using the KICA‐cog suggested a sensitivity ranging from 91% to 93% and specificity ranging from 90% to 94% when identifying dementia in different samples of Indigenous peoples in Australia. Other commonly used cognitive screening tools such as the MMSE suggested sensitivity of 85% and specificity of 82% in community‐based predominantly White samples in one diagnostic test accuracy meta‐analysis and 81% sensitivity and 89% specificity across more than 100 studies and a variety of samples. Classification accuracy for identifying MCI was lower for the CICA, where sensitivity was 97% but specificity was 40% (cut‐point of 37/39), which is lower than the classification accuracy for identifying dementia. A similar drop in sensitivity is seen for other screening tests such as the MMSE, where a meta‐analysis of identification of MCI with the MMSE suggested sensitivity of 66% and a specificity of 74% across majority culture samples. The classification accuracy for the CICA with Indigenous peoples is comparable to other screening tests used with predominantly White samples, and with the KICA when used with Indigenous peoples in Australia.

Measurement of cognition can vary for cognitive test items (i.e., differential item functioning) or for latent constructs (i.e., language demonstrating measurement invariance) based on cultural, language, and educational groups (see Pedraza and Mungas for a review), which is referred to as measurement bias. The important work in adapting and developing the CICA mitigated cultural biases in the design and approach to cognitive assessment and presents a culturally appropriate tool that is suitable for clinical use measuring cognition with Indigenous people, within and beyond Canada.

Future research on the CICA will be needed to provide data on the measurement equivalence of cognition measured with the CICA in English and in Anishinaabemowin. Translation of the CICA into additional languages and contexts of Indigenous groups will determine whether measurement invariance is stable, or whether larger studies will be necessary. Future work will also need to establish strong evidence for stability in the CICA's measurement (test‐retest reliability) to develop reliable change formulae, which are statistical methods to account for error in the CICA's assessment of cognition over multiple assessments. Ongoing and future reliability testing of the CICA would benefit from distinct inter‐rater and test‐retest methodology. While the study described in this article had two assessors evaluate each of the 15 subjects using the CICA (inter‐rater reliability), the testing was also done at different times on the same day. While this allowed for concurrent inter‐rater and intra‐rater reliability testing, future testing should consider successive testing in which either the assessor or the time of day is varied. It is possible that the second administration of the CICA on the same day may have been influenced by a practice effect on the part of the participant or influenced by the time of day. We will need to compare changes in CICA scores over time to observed declines in function to establish minimum clinically important differences in scores and to support the CICA's utility in the measurement of cognitive decline.

Recruitment into the study was facilitated by our CBPR approach. Attrition was impacted by having only 6 days within a 1‐week period available with the geriatricians for the clinical assessment. In retrospect, over‐recruitment would be advisable with this approach. Because attrition was based on weather, community events that conflicted with appointments, or due to loss of community members due to death, we expect that attrition was random, and would have been equally likely in those who would have been diagnosed with no cognitive impairment, MCI, or dementia and would have no impact on the classification accuracy.

Our mixture of snowball and purposive sampling methods is a starting point; however, further evidence for validity of the CICA should be demonstrated using population‐based sampling procedures, a method that usually results in more clinically useful estimates of classification accuracy. With our current sample size in the validity study, the power to determine cut‐off for MCI was only 0.67. Thus, the promising results of the present study would be supported by future validation studies that used larger samples, both population‐level and clinical, and other Indigenous populations in diverse settings.

This important first step lays a foundation for the development of a suite of culturally safe assessment tools for older Indigenous people in Canada. Notably, the validation process highlighted the need for corroborative information provided by knowledgeable caregivers, signaling the potential for a tool to use with caregivers that would inform a functional assessment. This need was identified by physicians during the validation component and highlighted by our community advisory council as key to appropriate cognitive assessment.

In conclusion, the CICA was developed in ways that are responsive to the needs of First Nations community members, leadership, and health‐care providers with the intent to improve dementia care for older First Nations people, care partners, and communities. This important community‐engaged research has resulted in a First Nations‐specific, culturally safe, trauma‐informed, reliable, and valid cognitive assessment to support dementia case‐finding.

CONFLICTS OF INTEREST

The authors report no conflicts of interest.

AUTHOR CONTRIBUTIONS

Jennifer D. Walker, PhD, was the principal investigator for the validation study, co‐led the statistical design of the study, supervised data analysis, and wrote the paper. Megan E. O'Connell, PhD, co‐investigator of the study, co‐led the statistical design and data collection procedures for the study, assisted in analyses, and assisted in writing the paper. Karen Pitawanakwat, RN, primary community liaison, collected the data, supervised the community approval process, and assisted in writing the paper. Melissa Blind, PhD, co‐investigator of the study, project management, assisted with community engagement, supervised and assisted with data collection, and assisted in writing the paper. Wayne Warry, PhD, co‐investigator of the study, contributed to study design, and assisted in writing the paper. Andrine Lemieux, PhD, assisted in writing the paper. Christopher Patterson, MD, performed cognitive assessments and assisted in writing the paper. Cheryl Allaby, MD, performed cognitive assessments, and assisted in writing the paper. Yantao Zhao, MSc, conducted the data analysis and assisted in writing the paper. Kristen Jacklin, PhD, was the principal investigator for the reliability study, co‐principal investigator for the validity study, conceived of and designed the study, led community engagement, supervised data collection, and assisted in writing the paper.

Supporting Information

Click here for additional data file.