Development and validity of the Computer-Based Cognitive Assessment Tool for intervention in community-dwelling older individuals.

AIM: The Mini-Mental State Examination is a widely used cognitive assessment tool. However, it has several limitations, including the learning effect and interrater reliability. Therefore, we developed a Computer-Based Cognitive Assessment Tool (CompBased-CAT), which runs on a tablet or personal computer. In this study, we examined the validity and discrimination ability of the CompBased-CAT.
METHODS: Participants were recruited from the Otasha-Kenshin study carried out in 2016. We included 773 community-dwelling older individuals in Japan (332 men, 441 women, aged 65-97 years). CompBased-CAT scores were converted to z-scores, and the correlation with Mini-Mental State Examination scores was examined using Pearson's correlation coefficient. Furthermore, the ability to discern cognitive impairment was examined using the receiver operating characteristic curve.
RESULTS: The Pearson's correlation coefficient for the Mini-Mental State Examination scores and each task component of the CompBased-CAT ranged from 0.24 to 0.41 (P < 0.001), and the correlation coefficient of the total z-scores was 0.51 (P < 0.001). The sensitivity, specificity and area under the receiver operating characteristic curve of the discriminating ability of the CompBased-CATool for cognitive impairment were 0.81, 0.77 and 0.85, respectively.
CONCLUSIONS: The CompBased-CAT certainly possesses validity, discriminating ability and utility as a new cognitive assessment tool in community-dwelling older individuals. Geriatr Gerontol Int 2020; ••: ••-••.

Introduction

The incidence of dementia has increased due to the rapidly aging population.1 It is estimated that the global number of patients with dementia will reach 115.4 million by 2050.2 Dementia is related to a decline in quality of life, and increased economic and patient care burdens.2

Unfortunately, current drugs are unable to prevent or cure dementia.3, 4 Therefore, early detection of cognitive decline is essential to prevent dementia. Early detection can effectively identify the disease in its treatable stage and provide lifestyle guidance in an attempt to slow cognitive impairment. This will allow patients and their families to adequately prepare themselves if the patients are at risk for developing dementia.5 In recent years, tests such as neuroimaging and genetic biomarkers have been developed. However, they are expensive and not readily available to community‐dwelling older individuals.6

The Mini‐Mental State Examination (MMSE) is a widely used cognitive assessment test, which screens for functional cognitive impairment and tracks changes in cognitive function over time.7, 8 However, the MMSE, which consists of a series of questionnaires, has several limitations, including a learning effect from repetitive measurements, interrater reliability and the need for several trained personnel for its administration.9 Furthermore, the MMSE primarily assesses memory and language function, and shows poor sensitivity toward detecting frontal lobe dysfunction.9

Therefore, we developed a Computer‐Based Cognitive Assessment Tool (CompBased‐CAT) consisting of six tasks, that runs on a tablet computer (PC), which automatically executes the test.

In the present study, we first examined the relationship between the CompBased‐CAT and MMSE scores to verify the validity of the CompBased‐CAT.

Methods

Participants

Participants were recruited from comprehensive health checks that were part of the Otasha‐Kenshin study carried out in 2016 at the Tokyo Metropolitan Geriatric Hospital and Institute of Gerontology (Tokyo, Japan).10, 11 The inclusion criteria were older community‐dwelling individuals, aged ≥65 years. We excluded 54 participants, who did not consent to undergo cognitive assessment and those who found the CompBased‐CAT difficult. Thus, data from 773 individuals were analyzed. All participants provided informed consent. This study was approved by the ethics committee of the Tokyo Metropolitan Institute of Gerontology (approval number H14,2016).

Cognitive measurements

Measurement of the CompBased‐CAT

The CompBased‐CAT was created to assess specific cognitive functions. It consisted of the following six tasks (Fig. 1): (i) digit span forward (attention and concentration)12 (ii) digit span backward (attention and concentration)12 (iii) memory of item names (immediate memory); (iv) memory recall of item names (remote memory); (v) Stroop (executive function and selective attention)13 and (vi) recognition of figures (space perception). This test was carried out using a tablet PC (operating system: Windows 10; ASUS TransBook T100HA, ASUS, Taipei, Taiwan, China,). Participants wore noise‐canceling headphones and responded to all tasks according to the instructions of the computer. Each test included one practice trial, and individuals were instructed to respond each task after practicing.

Figure 1

The Computer‐Based Cognitive Assessment Tool tasks. (a) Task 1 and 2: digit span forward and backward tasks. (b) Tasks 3 and 4: memory and recall of item names tasks. (c) Task 5: Stroop task. (d) Task 6: recognition of figures.

Tasks 1 and 2: Digit span forward and backward tasks

In task 1, participants were required to immediately recall a set of numbers that were presented in random order on the PC screen. In task 2, participants recalled the same set of numbers in reverse order. The number of digits gradually increased (3, 5, 7, 8). The maximum number of digits that a participant could recall in the correct order (forward or reverse) was recorded.

Tasks 3 and 4: Memory and recall of item names tasks

In task 3, participants were instructed to memorize 10 target words that were presented on the PC screen. Subsequently, 20 words, including 10 target and 10 distracter words, were displayed, and participants were required to select the 10 target words. This was repeated for two trials and the number of correct answers on the second attempt was recorded. Furthermore, participants were instructed to recall the 10 target words, after all other tests, and the total number of target words recalled was recorded.

Task 5: Stroop task

In task 5, the names of colors were displayed in an incongruent fashion (for example, “aka” [red in English] was written in blue font) on the PC screen. Participants were required to say the color of the font and not read the word, among four options (the correct answer for the above would be blue). The total number of correct answers for each of the 20 items in the task was recorded.

Task 6: Figure recognition task

In task 6, blocks were stacked sterically and displayed on the PC screen. Participants were asked to state the number of the blocks. We recorded the total number of correct answers for each of the eight items in the task.

Measurement of the MMSE

We used the Japanese version of the MMSE, which measures cognitive function in both clinical and research settings, to assess the validity of the CompBased‐CAT.14 This test was administered by trained and registered clinical psychologists. The MMSE and Comp‐Based CAT were assessed on the same day.

Other measurements

We included age at enrollment, sex, health status and educational level as demographic variables; body mass index and maximum gait speed as physical factors; the simplified Japanese version of the World Health Organization‐Five Well‐Being Index as a psychosocial factor;15 and the Tokyo Metropolitan Institute of Gerontology Index of Competence as an instrumental activity of daily living factor.16 We also assessed the independence in carrying out five basic activities of daily living (bathing, dressing, walking, eating and continence).

Statistical analysis

CompBased‐CAT scores were converted into z‐scores. We calculated the total score for each individual as the CompBased‐CAT score. We used Pearson's correlation coefficient to compare the CompBased‐CAT and MMSE scores. Student's t‐test and Pearson's χ2‐test were used to compare the characteristics between individuals with and without cognitive impairment in accordance with earlier criteria (cognitive impairment absent: MMSE score ≥24, cognitive impairment present: MMSE score <24).17 Furthermore, we used receiver operating characteristic (ROC) curve analysis to determine the cut‐off CompBased‐CAT score that was equivalent to an MMSE score <24. We calculated the cut‐off point using the Youden Index. The area under the ROC curve for the CompBased‐CAT score was used to measure its accuracy of discrimination between individuals with and without cognitive impairment. All analyses were carried out using IBM spss version 21.0 (IBM Japan, Tokyo, Japan). Statistical significance was set at P < 0.05.

Results

The characteristics of the participants are presented in Table 1. The mean age (SD) was 72.6 years (6.4 years), the range was 65–89 years and 441 participants were women (57.1%). The mean duration of schooling (SD) was 12.8 years (2.7 years), and the mean MMSE score (SD) was 28.3 (1.9). A total of 27 (3.4%) participants were assessed to have cognitive impairment. An overwhelming majority of participants (98.6%) could independently carry out activities of daily living. The original score and z‐score of the CompBased‐CAT are presented in Table 2. The z‐score of each item and total z‐score of participants with cognitive impairment (total z‐score: mean −6.04, SD 6.34) were significantly lower than those for participants without cognitive impairment (total z‐score: mean 0.19, SD 3.03). Table 3 presents the Pearson's correlation coefficient between the MMSE scores and each task component of the CompBased‐CAT, which ranged from 0.24 to 0.41 (P < 0.001). The correlation coefficient between the MMSE and CompBased‐CAT total scores was 0.51 (P < 0.001). Figure 2 shows the ROC curve of the total z‐score of the CompBased‐CAT, which discerns the presence of cognitive impairment (MMSE <24). The area under the ROC curve sensitivity and specificity were 0.85, 0.81 and 0.77, respectively.

Table 1

Participants’ characteristics

Variable	Mean (SD) or n (%)
Age (years)	72.6 (6.4)
Women, n (%)	441 (57.1)
Health status, n (%)
Cerebral stroke	48 (6.2)
Heart disease	110 (14.2)
COPD	11 (1.4)
Diabetes mellitus	98 (12.7)
Depression	32 (4.1)
Education (years)	12.8 (2.7)
BMI (kg/m2)	23.0 (3.4)
Max gait speed (m/s)	2.1 (0.4)
WHO‐5	16.3 (5.0)
TMIG‐IC	12.3 (1.2)
MMSE	28.3 (1.9)

BMI, body mass index; COPD, chronic obstructive pulmonary disease; MMSE, Mini‐Mental State Examination; TMIG‐IC, Tokyo Metropolitan Institute of Gerontology Index of Competence; WHO‐5, World Health Organization‐Five Well‐Being Index.

Table 2

Comparison between the original scores and z‐scores of the Computer‐Based Cognitive Assessment Tool of participants with and without cognitive impairment

	Scale range	All	MMSE ≥24	MMSE <24	P‐value
		n = 773	n = 746	n = 27
Digit span forward	0–8	5.3 (1.4)	5.4 (1.4)	3.9 (1.6)	<0.01
			0.04 (0.97)	−1.04 (1.16)	<0.01
Digit span backward	0–8	4.5 (1.5)	4.5 (1.5)	3.1 (1.4)	<0.01
			0.03 (0.99)	−0.91 (0.92)	<0.01
Memory of the item name	0–10	8.9 (1.2)	9.0 (1.1 )	7.0 (2.4)	<0.01
			0.06 (0.90)	−1.55 (2.00)	<0.01
Memory recall of the item name	0–10	7.1 (3.4)	7.2 (3.3)	3.4 (3.9)	<0.01
			−0.04 (0.26)	−0.33 (0.30)	<0.01
Stroop task	0–20	2.3 (1.5)	15.9 (6.0)	10.0 (6.4)	<0.01
			0.32 (9.83)	−9.22 (10.5)	<0.01
Recognition of figures	0–6	4.7 (1.6)	4.7 (1.6)	3.8 (1.8)	0.02
			0.03 (0.98)	−0.52 (1.10 )	<0.01
Total z‐score	NA	NA	NA	NA
			0.19 (3.03)	−6.04 (6.34)	<0.01

The top row shows the original scores and the bottom row shows the z‐scores. NA, not available.

Table 3

Pearson's correlation coefficients of the Mini‐Mental State Examination and z‐scores of each task component of the Computer‐Based Cognitive Assessment Tool

	Correlation coefficient
	r	P
Digit span forward	0.35	<0.001
Digit span backward	0.33	<0.001
Memory of the item name	0.41	<0.001
Memory recall of the item name	0.30	<0.001
Stroop task	0.28	<0.001
Recognition of the figure	0.24	<0.001
Total z‐score	0.51	<0.001

Figure 2

Receiver operating characteristics curve of the total score of the discrimination ability of the Computer‐Based Cognitive Assessment Tool for assessing cognitive impairment (Mini‐Mental State Examination score <24). AUC, area under the curve; CI, confidence interval.

Discussion

The present study developed the CompBased‐CAT, and investigated its validity and accuracy for assessing cognitive impairment. We found a moderate correlation between the CompBased‐CAT total and MMSE scores. Furthermore, the CompBased‐CAT showed high sensitivity and specificity for detecting cognitive impairment.

The characteristics of the participants of the present study, which included MMSE scores, the prevalence of cognitive impairment, educational history, maximum walking speed and Tokyo Metropolitan Institute of Gerontology Index of Competence, were similar to other large‐scale cohort studies carried out in Japan.18, 19, 20 Therefore, the samples in this study were regarded as representative of community‐dwelling older individuals in Japan.

The original CompBased‐CAT score, z‐score of each item and total z‐score, which was calculated from the z‐score of each item, were significantly lower in those individuals with cognitive impairment than those without, indicating that CompBased‐CAT reflects cognitive function. As each domain of the CompBased‐CAT has a different range of scores, we used the z‐score for each item to calculate the total z‐score and to make comparisons between the domains. The scores of the item‐memory and Stroop tasks showed a remarkable difference between the cognitive impairment and non‐cognitive impairment groups. It was suggested that the decline in the scores of the item‐memory and Stroop tasks was pronounced in patients with cognitive impairment. This finding is consistent with those of earlier studies.21

The correlation coefficient between the z‐score of each domain and the MMSE score was approximately 0.24–0.41, whereas the correlation coefficient between the total z‐score and the MMSE score was the highest at 0.51, suggesting that the total z‐score is more effective in discriminating cognitive impairment than the z‐score for each domain alone.

Previous studies have reported the discrimination ability of computer‐based cognitive tests, with sensitivity ranging from 0.68 to 1.00, specificity ranging from 0.73 to 0.98 and the area under the ROC curve ranging from 0.80 to 0.99.22 The discrimination ability of the CompBased‐CAT was similar to these previous studies, which indicates that the CompBased‐CAT possesses validity and discrimination ability as a cognitive test.

Several computer‐based cognitive assessment tools were developed in earlier studies and showed a correlation with established neurocognitive measurements.23, 24, 25 However, the sample size of those studies was very small, and only included individuals who visited clinics or had no cognitive problems. Therefore, they might not be representative of community‐dwelling older individuals in Japan. In the present study, we recruited a large number of individuals with a wide range of MMSE scores that were highly representative of the general older adult population in the community, and showed that CompBased‐CAT has high validity in community‐dwelling older people in Japan. Furthermore, the scores reported in this study can be used for reference among community‐dwelling older individuals.

The CompBased‐CAT has several advantages over the MMSE. First, the CompBased‐CAT tasks are randomized by the PC; therefore, the CompBased‐CAT can eliminate any learning effect. Interestingly, an earlier study shows that there are few learning effects associated with randomizing problems.9 Second, the CompBased‐CAT is measured automatically by the PC; therefore, there is no instruction bias or discrepancy due to personal judgment26 and multiple personnel with specialized knowledge are not required to carry out the test. Third, the measurements are recorded, and the scores calculated automatically, which eliminates human error and reduces examination time, thus providing immediate results. In addition, the CompBased‐CAT enables us to administer the test and calculate the scores in 10–15 min. The MMSE reportedly takes approximately 10 min (not including the time required for scoring) if the examination is carried out by a skilled physician.27 More time is required when unskilled examiners carry out the examination or, especially, calculate the scores. Thus, the CompBased‐CAT requires approximately 10 min less than the conventional MMSE test to score and record results. Therefore, CompBased‐CAT is beneficial in terms of its time efficiency, and useful for evaluating a large number of participants during community health checkups while providing immediate feedback to the participants.

Finally, the MMSE primarily assesses memory and language function, whereas the CompBased‐CAT includes a task that assesses extensive cognitive function, which includes memory and language. Therefore, the CompBased‐CAT can measure cognitive function in a more comprehensive manner, which gives it greater utility compared with the MMSE.

The present study had the following limitations. First, our working definition of cognitive impairment was biased, as we defined it using the MMSE, and a definitive diagnosis of dementia was not established. The main purpose of this study was to determine the efficacy of the CompBased‐CAT for screening functional cognitive impairment; therefore, we considered the MMSE, which assesses cognitive function, to be an appropriate test for comparison. Second, we were unable to determine the reliability of the CompBased‐CAT. Earlier studies have shown that similar computer‐based cognitive tests have a high test–retest reliability.22 However, it is necessary to verify the reliability of CompBased‐CAT in the future. Finally, this was a cross‐sectional study, and a longitudinal study is necessary to examine the ability of the CompBased‐CAT to predict the onset of dementia.

In summary, we showed the validity and the discrimination ability of the CompBased‐CAT for assessing cognitive dysfunction. We also showed its utility for assessing community‐dwelling older individuals in Japan. We believe that using the CompBased‐CAT in a large‐scale cohort study or local health checkup could help prevent the onset of dementia by screening individuals with a high risk of dementia. This tool is available only in Japanese; thus, the development of an English version in addition to the verification of its validity and reliability are issues that need to be approached in the future. However, it is assumed that the need for evaluating cognitive function through simple tests will continue to increase in the countries that have aging populations. Therefore, the present study is relevant, as it shows the validity of computer‐based cognitive function tests in community‐dwelling older individuals.

Disclosure statement

The authors declare no conflict of interest.