Psychometric Properties of the Montreal Cognitive Assessment (MoCA) to Detect Major Neurocognitive Disorder Among Older People in Ethiopia: A Validation Study.

Background: Cognition is one of the most complex functions of the human brain, and major neurocognitive disorders affect this function causing a wide array of problems in an individual's life. Screening for major neurocognitive disorders can be helpful in designing and implementing early interventions. Purpose: This study was designed to assess the reliability and validity of the Montreal Cognitive Assessment (MoCA) tool to detect major neurocognitive disorders among older people in Ethiopia.
Methods: One hundred and sixteen randomly selected older adults in Ethiopia were involved in a cross-sectional study. The Diagnostic and Statistical Manual of Mental Disorders criteria for major neurocognitive disorders was used as a gold standard. Data were analyzed using STATA v16 statistical software. Receiver operating curve analysis was performed, and inter-rater, internal consistency reliabilities, content, criterion and construct validities were determined. Statistically significance was declared at a p-value of <0.05.
Results: The study had a 100% response rate. The mean age of the study participants was 69.87 ± 7.8. The inter-rater reliability value was 0.96, and Cronbach's alpha was 0.79. The optimal cutoff value was ≤21, and Montreal Cognitive Assessment has an area under curve value of 0.89. The sensitivity, specificity, positive and negative likelihood ratios, and positive and negative predictive values of MoCA are 87.18%, 74.03%, 3.35, 0.17, 63%, and 91.9%, respectively. The tool also has good concurrent and construct validities.
Conclusion: The Montreal Cognitive Assessment tool was a reliable and valid tool to detect major neurocognitive disorder. It can be incorporated into the clinical and research practices in developing countries.

Introduction

Cognition is one of the most complex functions of the human brain that incorporates language, attention, orientation, memory, judgment, abstract reasoning, problem-solving, praxis, and executive function.1

Major neurocognitive disorder (Dementia) is a neurodegenerative disorder characterized mainly by progressive decline in the cognitive functioning.1–3 Globally, it is estimated that around 55 million people live with Dementia, which is the seventh leading cause of mortality worldwide.4 A study in Ethiopia reported a 43.8% prevalence of Dementia among older adults among whom22.9% were reported to have moderate-to-severe cognitive impairment.5

Effective diagnosis and identification of the disorder and its treatment require extensive evaluations, diagnostic procedures, imaging, and laboratory tests. However, all these may not always be available, especially in resource limited settings. For this reason, brief and effective screening tools have been developed over the years.4

The need for such tools is even more pronounced in the developing countries with a significant gap in human, material, and facility resources to effectively identify those with the dementia.3,6 Availability of such the screening tools would enable service providers in Low- and Middle-Income Countries (LMIC) to identify the disorder and direct effective interventions.3,4,7

Many of the currently available cognitive screening tools were developed in the socioeconomically developed settings.4,8 This warrants the development of new tools or testing the available ones for their applicability in low educated and highly diverse populations of LMICs.

MoCA is a brief cognitive function assessment tool with 12 items developed by Nasreddine et al in 2005.9 Visuospatial, executive functioning, short-term memory, attention/concentration/working memory, language, and orientation are the cognitive domains assessed by the tool.9–11 A cut point of 26 has been recommended to identify major neurocognitive disorder and MoCA had 100% sensitivity, 87% specificity, 89% positive predictive value, and 91% negative predictive values in the initial validation study.9

Studies have evaluated the effectiveness of MoCA in different settings around the world, and it has been shown to be valid and reliable tool to detect dementia.9,12 The tool has also been especially praised for its ability to detect subtle changes in the cognitive functioning.7,11 It has been tested in several settings, including some countries in Africa, and evidence is available for its validity and reliability.13,14 Studies have also reported MoCA to be relatively resistant to linguistic and cultural variabilities even though it was reported to be affected by educational status.9 This has led to a recommendation for a one-point correction for those with lower levels of education which was indicated to dissolve the education effect in several studies.15,16

Even though a wealth of evidence is available for its effectiveness in various setups and detecting cognitive impairments resulting from a range of etiologies, few validation studies have been done in Africa. As to the researcher’s best knowledge, it has never been validated in Ethiopia.

Ethiopia is a unique nation situated in East Africa with low coverage and access to education and high level of illiteracy.17 The country is also known for its cultural and linguistic diversity. Testing the applicability and validity of such cognitive screening instruments in the country is therefore critical to guide evidence-based decision-making. This study was therefore designed to determine the psychometric properties of MoCA among older people in Ethiopia to detect major neurocognitive disorder and determine factors associated with it.

Methods

An institution-based cross-sectional study was conducted between August – September, 2020 among older people living in a non-for-profit residential center in Addis Ababa, Ethiopia. Age greater than or equal to 60 and residential status within the center were the inclusion criteria for the study. The study was part of a larger study, and a further description of the methods can be found in another publication by the same authors.18

Sample Size Determination

We followed the procedures for calculating sample size to determine the diagnostic accuracy of a diagnostic test or AUC. Further description of the procedures can be found elsewhere.19 For the calculation of the sample size, alpha was set to be 0.05, power set at 90%, 1:2 ratio between positive and negative cases, and 0.5 was taken as a null value. The expected AUC was 0.7, taken from a study done in Tanzania in 2020 for the AUC of MoCA and the calculated sample size was 105.13 A 10% non-response rate was added, and the final sample size was 116. The sample size calculations were performed using MedCalc Version 19.1.3 software. Simple random sampling was employed to select study participants after generating a sampling frame using a list of individuals living in the center.

Data Collection Instruments

Socio-demographic characteristics that included age, sex, marital status, ethnicity, area of residence, and educational status were collected with a brief questionnaire.

Montreal Cognitive Assessment Tool (MoCA)

The Montreal Cognitive Assessment (MoCA) is a one-page, 30-point cognitive screening measurement scale that takes about 10 minutes to administer. There are 12 subtasks in the MoCA test that include memory (recall of five nouns, after two verbal presentations) (5 points); visuospatial orientation (clock-drawing (3 points) and cube copying (1 point)); executive functioning (trail making B task (1 point), phonemic fluency (1 point), and two-item abstract thinking task (2 points)); attention (target tapping; 1 point), concentration (serial subtraction task (3 points)), and working memory (forward and backward digit span (1 point each)); language (naming animals of low-familiarity (3 points), sentences repetition (2 points) and the phonemic fluency task presented above; orientation to time and place (year, month, date and day of the week as well as the place and the city) (6 points).20 Developers have recommended a score of 26 as a cutoff score to differentiate between normal and abnormal.2,9

Gold Standard Assessment

The American Psychiatric Association’s (APA) Diagnostic and Statistical Manual fifth edition (DSM 5) criteria for Dementia was used as a gold standard diagnosis.1 The criteria require a progressive decline in cognitive functioning, interference with independence in daily living and exclusion of other possible causes. As part of the gold standard diagnostic evaluation, the Mini-Mental State Examination (MMSE) was used as a standardized cognitive assessment instrument (Criteria A2).1 Impairment in the daily activities was measured by the data collectors during the DSM assessment by asking questions regarding the performance of the participants on basic and instrumental activities of daily living. A full description of the diagnostic criteria can be referred to from the APA’s manual or in other publications.1,18

Geriatrics Depression Scale (GDS) short form, a 15-item self-report measure by Yesavage et al with Yes/No questions, was used to assess depression.21,22 The tool has also been applied among the Ethiopian older adults and was reported to have good internal consistency.23

Data Collection Procedures

Two BSc nurses and two BSc psychiatry professionals with at least two years of clinical experience and with previous experience of collecting research data were used as data collectors. The data collectors were provided with a one-day training by the research team members prior to data collection. The BSc nurses applied MoCA, and the psychiatry professionals conducted the gold standard assessment. A one-week period was taken between the assessment by the gold standard and MoCA for a single participant. The two groups of data collectors were blinded to the performances of the test in the other assessments. To prevent the order of test effect, the order of the two tests was interchanged for every case.

Inter-Rater Reliability

Two data collectors interviewed 20 individuals using the tool to determine its inter-rater reliability. The involved interviewers were blinded to the interview findings of each other.

Translation and back-translation procedure was applied to translate all the questionnaires into the Amharic language. Bilingual experts with experience in translation of research instruments independently translated the questionnaires from English to Amharic then back to English. The three versions, ie, the original English, the Amharic, and the back translated versions of the questionnaires were carefully reviewed and any discrepancies were resolved after discussion with the involved professionals.

MoCA is generally translated to Amharic without major difficulties. However, some of the items in the tool required adaptation into the Amharic language. The first is the trail-making test (TMT) in the visuo-constructional skills question. The English letters A - E were replaced with the first five letters in the Amharic language “ሀ፤ ለ፤ ሐ፤ መ፤ ሠ”. Similarly, the English letters in the letter tapping test were replaced by their phonemic equivalences in the Amharic alphabet except “A”, which was replaced by the first letter in the Amharic alphabet “ሀ”.

Another modification was made on the verbal fluency item. Letter “B” was replaced with its phonemic consonant equivalence “በ”. Furthermore, the term “screwdriver” in the abstraction question, which has an Amharic language meaning of “Yetemenja Mefcha”, was deemed to be a less familiar item, and therefore the question was replaced with “Awtobus ena babur”, which means “Bus and Train”. The other items, “Matches and Lamp”, with their direct Amharic meanings “Kibrit ena ampole” had little connections in the Amharic language and therefore “matches” was replaced by “Shama”, which meant “Candle”.

Before the data collection, one-day training was provided for data collectors and supervisors on the instruments, ethical principles, and the way they diagnose cases.

Data Processing and Analysis

The data, after collection were coded and entered into Epi data entry V. 4.6.1 software and the analysis was performed using STATA V.16. Internal consistency reliability was measured using Cronbach’s alpha coefficient. Inter-rater reliability was determined using the intra-class correlation coefficient.

The content validity of MoCA was determined in two ways. The first was determining the Content Validity Index (CVI). A panel of nine members was selected from senior professionals with knowledge and experience in the fields of Psychiatry (6 experts), one epidemiologist, and two neurologists. Each panelist rated each item in the tool from 1 to 4 considering their clarity and relevance. Item- and scale-level CVIs were then computed. Explanations for the computation of the indices are available elsewhere and can be found in the literature.24

As a second method of content validation, a well-trained data collector interviewed 20 sampled individuals, dividing them into two halves. During the interviews, respondents were asked to elaborate and explain their understanding of each question and their answers. Respondents were also asked for suggestions on how to replace better fitting words, if any. Whenever the meaning of the items was not clear, when the responds faced difficulties in elaborating or when the questions were wrongly understood, the respective items were modified. The updated version was then administered to the second half of the participants.

Receiver operating curve (ROC) analysis was performed to test diagnostic accuracy. At different cutoff points, the specificity, sensitivity, positive and negative predictive values, and positive and negative likelihood ratios of MoCA were determined. The optimal cutoff value was determined using the Youden’s J index (sensitivity + specificity−1). Pearson’s correlation coefficient was used to test concurrent validity by determining the correlation between MoCA and MMSE. Generally, a correlation greater than or equal to 0.7 is recommended to indicate good concurrent validity. However, McDowell et al argue that a correlation greater than or equal to 0.6 should be sufficient to indicate strong correlation taking into account the unreliability associated with the scales due to the attenuation of validity coefficients.25 An independent sample t-test was used to evaluate the construct validity of MoCA by applying the known group validity approach. The approach was tested to see if there was a significant mean MoCA score difference between the dementia and non-dementia groups based on the gold standard assessment. P value <0.05 was used to declare statistical significance.

The association between the scores of MoCA and different factors was evaluated using the multiple linear regression analysis. All assumptions of multiple linear regression were checked, and statistical significance was declared at P < 0.05.

Results

Socio Demographic Characteristics

The study has a 100% response rate with a total of 116 respondents. Males comprised 51.72% (n = 60) of the participants and the mean age of the participants was 69.87 ± 7.97 (Range; 60–94). The respondents’ mean years for formal education were 4.90 ± 5.90 years. The majority (36.2%) of the participants had Amharic as their mother tongue, followed by Affan Oromo (35.3%). Before their admission to the center, 55.17% (n = 64) of the respondents had their residence in urban areas. Other sociodemographic characteristics of the respondents can be observed in Table 1.

Table 1

Socio-Demographic and Clinical Characteristics of the Study Respondents at Macedonia Humanitarian Center, 2020

Variable	Category	Frequency	Percent	Mean (95% CI) MoCA
Age	60–64	34	29.31	22.68 (21.0–24.3)
	65–69	32	27.59	19.53 (17.3–21.8)
	70–74	27	23.28	17.89 (15.5–20.3)
	75 and above	23	19.82	20.87 (18.6–23.1)
Sex	Male	60	51.72	21.31 (20.1–22.5)
	Female	56	48.28	19.28 (17.4–21.1)
Mother tongue	Amharic	42	36.21	20.11 (18.3–21.9)
	Affan Oromo	41	35.34	20.44 (18.3–22.6)
	Tigrigna	15	12.93	21.72 (19.5–23.9)
	Guragigna	13	11.21	18.69 (16.1–21.3)
	Others#	5		21.4 (15.9–26.9)
Educational status	No formal education	56	48.80	17.3 (15.7–18.9)
	Primary Education	32	27.59	22 (20.3–23.7)
	Secondary Education	13	11.21	23.84 (22.2–25.5)
	College and above	15	12.93	25.0 (22.9–27.2)
Marital Status	Never Married	24	20.69	20.71 (18.5–22.9)
	Married	33	28.45	24.09 (22.4–25.8)
	Divorced	10	8.62	16.8 (12.7–20.9)
	Widowed/Widower	49	42.24	18.35 (16.7–19.9)
Area of residence before admission to the center	Urban	64	55.17	19.13 (17.4–20.9)
	Rural	52	44.83	21.31 (19.9–22.7)
Diagnosis (DSM-5)	Dementia	39	33.62	14.92 (13.5–24.1)
	No Dementia	77	66.38	23.08 (22.0–24.1)
Depression (GDS)	No depression	44	37.93	23.25 (21.9–24.5)
	Mild to moderate depression	61	52.59	18.66 (17.1–20.2)
	Severe depression	11	9.48	18 (13.8–22.2)
Total		116	100	20.34 (19.2–21.4)

Note: #Others = Sidamu, Wolitigna, Kaffigna.

Reliability of MOCA

The inter-rater reliability or an ICC value of MoCA was 0.96 (95% CI: 0.91–0.98). The internal consistency (Cronbach’s alpha) value of MoCA was 0.79. When they were deleted, none of the items in the tool resulted in an increment of its alpha value.

Validity of MoCA

Content Validity

Generally, the items and the instructions on the items on MoCA were understandable and clear. However, some of the respondents with no to low years of formal education faced difficulty performing some of the tasks, including the box drawing and clock drawing tests, the serial sevens subtraction, and digit span tests. Nevertheless, they were clear and easily answered by those with relatively better years of formal education. The contents of these tests (items) were not modified from the original English version because when originally designed, the tool took these performance variations with levels of education into consideration and allowed one-point correction for those with less than twelve years of formal education.

Item 2: On the animal naming test, nine of the ten respondents asked in the first round of the interview responded that they did not recognize the picture of Rhinoceros “Awuraris” and asked if it could be changed. A picture of a horse subsequently replaced it.

MoCA had an excellent CVI score. On I-CVI, three items, ie, Clock drawing test, vigilance (letter tapping test), and serial sevens test, had 89, and the rest had 100. The overall S-CVI for MoCA was 97, which showed an excellent rating by the experts for the validity of its content.

Criterion Related Validity

The study participants had a mean MoCA score of 20.33± 6, and 12 minutes was the average time taken to administer the tool on a single respondent. MoCA had demonstrated an excellent diagnostic accuracy having an AUC of 0.89 (95% CI: 0.83–0.95) (Figure 1).

Figure 1

ROC curve of MOCA for detection of major neurocognitive disorder, 2020.

The optimal cutoff score with the maximum Youden’s index value was 21, and at this cut-point, MoCA has specificity, sensitivity, LR+, LR -, PPV, and NPV values of 87.18%, 74.03%, 3.35, 0.17, 63%, and 91.9%, respectively (Table 2). At the cutoff score of 26, MoCA had an excellent sensitivity of 100% but a significantly low specificity of 23.38%.

Table 2

Psychometric Properties of MOCA for Detection of Major Neurocognitive Disorder, 2020

Cutoff (≤)	Sensitivity	Specificity	Correctly Classified	Youden’s J Index	LR+	LR-	PPV	NPV
19	82.05%	77.92%	79.31%	0.59	3.71	0.23	65.30%	89.60%
20	82.05%	76.62%	78.45%	0.58	3.51	0.23	64.00%	89.40%
21	87.18%	74.03%	78.45%	0.61	3.35	0.17	63.00%	91.90%
22	94.87%	62.34%	73.28%	0.57	2.51	0.08	56.10%	96.00%
23	100%	54.55%	69.83%	0.54	2.20	0.00	52.70%	100%
24	100%	45.45%	63.79%	0.45	1.83	0.00	48.10%	100%
25	100%	38.96%	59.48%	0.39	1.63	0.00	45.30%	100%
26	100%	23.38%	49.14%	0.23	1.30	0.00	39.80%	100%

Note: Bold values indicate the optimal cutoff score.

Construct and Concurrent Validity

An independent sample t-test analysis showed a statistically significant difference between dementia and non-dementia groups in their mean MoCA scores (mean diff. = 8.15, t = 8.96, p < 0.001). MoCA is also significantly and positively correlated with the MMSE (r = 0.612, P < 0.001).

Factors Associated with the Performance of the Test

Being diagnosed with major neurocognitive disorder was associated with a 6.9 points decrement in MoCA scores (β: −6.889 [95% CI: −8.792, −4.987], P < 0.001). In addition, a single-point decrease in the geriatric depression scale scores was associated with 0.25 points decrease in MoCA score (β: −0.258 [95% CI: −0.506, −0.009], P = 0.042). On the other hand, a one-year increase in attendance of formal education was associated with a 0.2-point increment in MoCA test scores (β: 0.203 [95% CI: 0.025–0.381], P = 0.026) (Table 3).

Table 3

Multiple Linear Regression Model Results for Factors Associated with MoCA Scores, 2020

	Coef.(β)	Sig.	95% Confidence Interval
			Lower Bound	Upper Bound
Years of Formal education	0.203	0.026	0.025	0.381
Age	−0.042	0.442	− 0.151	0.066
Dementia Diagnosis	− 6.889	<0.001	− 8.792	− 4.987
Total GDS score	− 0.258	0.042	−0.506	−0.009
Gender female	− 1.034	0.235	− 2.754	0.684
Mother tongue#
Affan Oromo	−0.460	0.635	−2.379	1.458
Tigrigna	1.280	0.337	−1.353	3.914
Guragigna	0.124	0.931	− 2.702	2.951
Other##	0.226	0.915	− 3.972	4.424
Rural Area of Residence	0.555	0.535	−1.212	2.323
_cons	26.391	0.000	17.840	34.943

Notes: Dependent variable: MOCA total score. Bold- indicates P-values <0.005. #Amharic as mother tongue taken as constant. ##Sidamu, Wolitigna, Kaffigna.

The variables in the model explained 50% of the variation observed in the dependent variable, ie MoCA score (R2 = 0.53, Adjusted R2 = 0.50, F = 11.75, p < 0.001).

Discussion

In the current study, we attempted to determine the psychometric properties of MoCA. The findings indicated the excellent reliability of the tool in consistently measuring cognitive function and the excellent ability of the tool to provide consistent measurements when used by different raters. The study also indicated that at an optimal cutoff value of ≤21, the tool has an AUROC value of 0.89. MoCA had excellent specificity, sensitivity, and predictive values at this cutoff point. Participants’ years of formal education, diagnosis of dementia, and depression were factors that were significantly associated with MoCA.

Several studies have been conducted over the years to examine the validity and reliability of MoCA in different languages, settings, and population groups. Most of these studies have reported different reliability measures, and the findings are primarily consistent with the findings of the current study. The initial study of MoCA conducted in Canada and another study in Egypt for the Arabic version of the tool reported Cronbach’s alpha values of 0.83.9,26 Another study in Tanzania also reported a Cronbach’s alpha value that is very close to the current study’s finding (0.78).27 These consistent reports indicate that the tool has excellent reliability in measuring the cognitive functioning of individuals and among different settings.

The AUC for MoCA reported in the current study is well beyond the minimum requirements for good diagnostic and screening tests. The optimal cutoff values and the sensitivity and specificity values in the current study were lower than the initial development study.9 The higher cutoff value of the tool on the other study can be expected as the study was conducted in a socioeconomically well-developed population. However, the findings of this study are consistent with other studies, such as the one conducted in New Zealand among a population from a multicultural background.15 The New Zealand study reported that the AUC for MoCA was 0.84, and the cut point was reported to be 20/21 while sensitivity and specificity were 78.3% and 82.98%, respectively. Other studies conducted in Africa, including South Africa, Tanzania, and Morocco, all corroborated with the current study’s findings regarding the effectiveness and validity of MoCA in multicultural and linguistically diverse populations, including Africa.13,14,28

The statistically significant difference in mean MoCA scores between the two diagnostic groups corroborated the evidence provided in studies conducted in Columbia,29 the United States,30 and North Africa14 for the construct validity of the tool in which all studies also reported a difference in MoCA scores among those with cognitive impairment and no cognitive impairment.

Despite the one-point correction, MoCA in the current study has been found to have a statistically significant association with years of formal education attended. This is supported by reports of another study in China.16 However, studies conducted in New Zealand15 and Central America29 contradicted this finding in which the one-point correction resulted in decrements of the education effect on their studies.

Overall, the findings of this study demonstrated the applicability of the tool and its effectiveness in identifying cognitive impairments and screening neurocognitive disorders. The demonstrated ability of the tool to assess the cognitive status of individuals and identify cognitive disorders of various etiologies in the current study and other studies indicates that the tool is a valid, reliable, and practical instrument for use in Ethiopia. The tool was also shown to be suitable for the often-busy outpatient setups and can be utilized by professionals without a background in psychiatry/mental health. As far to the researcher’s best knowledge, this is the first study to evaluate and report the validity of MoCA in Ethiopia to this date.

Limitations

The findings of this study could be better interpreted with some considerations. First, the test–retest reliability of MoCA was not assessed in the current study, and it is an essential measure of reliability to assess the ability of the tool to measure cognitive functioning over time. In addition, the Amharic language is variable in different regions of the country, and caution must be taken while using the tool.

Conclusion and Recommendation

The Montreal Cognitive Assessment has been tested in the Ethiopian elderly population and has been demonstrated to be reliable and valid in determining major cognitive disorder. MoCA can be incorporated into clinical practice in the country. Further studies may validate the tool in other languages, settings, and population groups. Further studies should also consider examining the test–retest reliability of MoCA.