Intra-Individual Variability in Cognitive Performance Can Befuddle the Study of Cognitive Impairments and Decline.

Using group-aggregated results and snapshot assessments of cognitive performance may prove problematic if the assessed construct shows substantial and rapid variation over time. To illustrate the significance of this issue, we analyzed cognitive performance data of ten older adults undergoing daily computerized cognitive assessments (CogState Brief Battery) for 36-93 days. In all cases, the day-to-day intra-individual variability was substantial when compared with group-level, between-person variability. This indicates that the results of studies using single snapshot assessments of cognitive functioning should be interpreted with caution. Additionally, group-aggregated measures of cognitive performance may not directly extrapolate to an individual.

INTRODUCTION

Cognitive performance can show substantial short-term fluctuations within persons [1, 2]. Consequently, single assessments of cognitive performance can yield erroneous inferences [2]. These fluctuations in cognitive performance are typically larger in people with neurological dysfunction and cognitive impairment [3-6] and are regarded as a potential tool in diagnosing central nervous system pathology [6-8]. This suggests that, particularly in people with cognitive impairments, high intra-individual variability in cognitive performance should be taken into account when assessing cognitive performance.

Nevertheless, many studies on cognitive impairment and cognitive decline have relied on single assessments of cognitive performance, which may re-duce their accuracy. Moreover, if the magnitude of this intra-individual variability differs between persons, a clinically meaningful change in cognitive performance may also differ between persons. Furthermore, when there is substantial intra-individual variability, findings from between-person associati-ons (i.e., group-aggregated results) are unlikely to apply to individuals [9-12]. Most studies in the cog-nitive aging literature have typically relied on group-aggregated results while making inferences about individual persons. One may wonder to what extent these group-aggregated results apply to persons whose intra-individual variability substantially ex-ceeds the between-person variability.

We aimed to identify the relative magnitude of intra-individual variability of cognitive performance in older people with suspected or established diagnosis of mild cognitive impairment (MCI) with depressive symptoms by comparing intra-individual variability with between-person variability of cognitive performance.

METHODS

Study design and participants

The ‘idiographic study on Cognition Affect and Sleep in the Elderly’ (i-CASE [13]), used a single-subject observational study design. Following the idiographic single-subject design, each participant was repeatedly assessed over time, and the time series of each participant were analyzed separately.

Ten participants, recruited from psychiatry and memory clinics, were eligible if they were≥60 years, had depressive symptoms defined as a major depressive disorder according to Diagnostic Statistical Manual (DSM) criteria or a 15-item Geriatric Depression Scale (GDS) score≥4, had cognitive impairments defined as MCI or mild dementia according to the multidisciplinary team of the memory clinic, a Mini-Mental State Examination (MMSE) score of < 25, or a Montreal Cognitive Assessment (MoCA) score of < 26. Exclusion criteria were a somatic disorder influencing short-term survival; moderate to severe dementia; a bipolar, psychotic, or addictive disorder during the past two years; not being able to participate; or being mentally incompetent to give consent for study participation. The study was approved by the institutional review board at the University Medical Center Groningen (code: 2013/019), and written informed consent was obtained from each study participant.

Cognitive performance assessment

Participants underwent a daily electronic cognitive assessment for 36 to 93 days, yielding 33 to 90 observations per person (Table 1). Participants were instructed to conduct them within one hour before bedtime in a comfortable and quiet environment with minimal distraction. The overall average interval between assessments was 1.1 days –each participant took on average 3.3 breaks between assessments that were longer than 1 day (range: 2–12 days). Cognitive performance was assessed with the CogState Brief Battery (CBB), a computerized cognitive test battery developed specifically for repeated testing of individuals in the absence of supervision and a valid measure of multiple cognitive domains with minimal practice effects [14, 15].

Table 1

Participant characteristics at the start of the study period and quantitative description of their participation

Participants				GDS	MMSE1	MOCA2	Diagnosis memory clinic6	#Days	#Observations	Breaks
ID*	Sex	Age	Edu							#4	Mean length5
1.	F	66	HP	07	NA	23	NA	53	50	2	1.5
2.	F	63	LV	11	29	19	MCI	36	33	3	1
3.	M	73	SV	37	NA	NA	MCI	62	62	0	–
4.	M	83	GS	5	NA	25	NA	93	90	3	1
5.	F	62	LV	11	NA	24	NA	63	61	2	1
7.	M	71	Uni	11	NA	25	SCI	63	61	1	2
8.	F	69	HP	6	NA	24	NA	62	50	10	1.2
9.	M	70	LV	10	NA	25	NA	63	49	33	4.7
10.	F	68	HP	8	30	NA	MCI	77	66	2	5.5
11.	M	61	SV	12	22	NA	MCI	70	47	7	3.3
Mean±SD		68.1±6.3	–	8±4		–	–	64.2±14.0	56.9±14.4	3.3±2.8	2.36±1.6

NA, not available; GDS, 15-item Geriatric Depression Score at the start of the study; MCI, mild cognitive impairment; SCI, subjective cognitive impairment; F, female; M, male; SD, standard deviation; HP, higher professional; LV, lower vocational; SV, secondary vocational; GS, general secondary; Uni, university. *Participant 6 was left out intentionally because this participant did not perform the cognitive tests 1Mini-Mental State Examination (MMSE) score at recruitment (0 = worst cognitive performance, 30 = best cognitive performance). 2Montreal Cognitive Assessment (MOCA) score at recruitment (0 = worst cognitive performance, 30 = best cognitive performance). 3There was 1 break of 12 days and 2 single-day breaks. 4Number of breaks between assessments that were longer than one day. 5Mean length of these breaks (in days). 6 Elaborated diagnosis of memory clinic for Participant 2 was somatoform delusion, primary psychiatric problems; for Participant 3, no further information was available because of transfer from other hospital; for Participant 7, no diagnosis or objective cognitive impairments; Participant 10 showed vascular MCI at baseline, but later had improved cognition and depression (2019); and Participant 11 showed cognitive impairments in more cognitive domains and diagnosed depression and psychotropic medication (benzodiazepines). Additional diagnostic research showed no signs of Alzheimer’s disease or cerebrovascular disease. Most likely, cognitive impairments were related to suboptimally treated depression and side effects of psychopharmacological treatment. 7 Although Participants 1 and 3 did not meet the inclusion criteria for depression (and were therefore excluded from the main study), we deemed them relevant to include in the present article.

CBB uses culturally independent game-like playing card tasks to assess psychomotor reaction time, attention, visual learning and memory, and working memory [16, 17]. The detection task (DET; psychomotor reaction time) asks to press the ‘yes’ button as quickly as possible if the playing card shown on the screen is turned face-up. The Identification Task (IDN; attention) asks to press the ‘yes’ button as quickly as possible if the card shown on the screen was red (and to do nothing if the card was black). The One Card Learning task (OCL; visual learning and memory) asks to press the ‘yes’ button if the card shown on the screen had been shown before during this trial and the ‘no’ button if not. The One Back Task (OBK; working memory) asks to press the ‘yes’ button as quickly as possible if the card shown on the screen was the same as the previous card, and the ‘no’ button if it was not. For DET and IDN, the base 10 logarithmic transformation of the reaction times for correct responses was the primary outcome. For OBK and OCL, the arcsine transformation of the proportions of correct responses was the primary outcome [16, 17].

Analysis

Intra-individual variability was estimated using the mean of squared successive differences (MSSD) [18] of each cognitive outcome for each person separately. The MSSD captures individual variability and accounts for temporal dependencies [19]. The MSSD is obtained by first calculating each successive time difference between assessments; then, each value is squared, and the result is averaged (see equation). In this way, the MSSD is not inflated by time trends, such as learning effects. t = number of assessments, i = consecutive occasions.

To assess the relative magnitude of intra-individual variability, we compared the square root of the MSSD (RMSSD) with the between-person SD of the same test. To estimate between-person variability, we used the between-person standard deviation (SD) of each cognitive outcome at Days 1 and 25. We chose SD on Day 1 as our analogy to the single assessment that most studies have used. Additionally, we chose SD at Day 25 to evaluate between-person variability after potential practice effects may have occurred. Day 25 was the last day of the study period on which all participants provided data. Furthermore, because ten participants were a relatively small sample to obtain a valid measure of between-person variability, we also retrieved the between-person mean and SD that were reported by Lim and colleagues [20] for these cognitive tasks in 47 people with MCI.

Our analyses were performed using R (ver. 14.0.3), the MSSDs were calculated using the psych package, and plots were generated with the ggplot2 package.

RESULTS

Participants

Five women and five men participated, 61–83 years old, with varying levels of education, and eight participants had elevated depressive symptoms (see Table 1). The highest relative intra-individual variability was found for the OCL, followed by the IDN and OBK, and DET had the lowest relative intra-individual variability (Fig. 1 and Table 2).

Fig. 1

Daily changes for each individual on each task relative to the between-person SD at Day 25. The between-person SD at Day 25 is indicated by the dotted lines. Dots above or below the dotted line indicate a daily change greater than the between-person SD. DET, detection task; IDN, identification task; OBK, one back task; OCL, one card learning task.

Table 2

Intra-individual and between-person mean and variability of the four cognitive outcomes

Intra-individual mean and variability
ID	DET		IDN		OCL		OBK
	mean	RMSSD	mean	RMSSD	mean	RMSSD	mean	RMSSD
1	2.537	0.044	2.706	0.036	0.838	0.150	1.351	0.248
2	2.725	0.066	2.833	0.069	0.649	0.150	0.955	0.192
3	2.729	0.088	2.902	0.048	0.613	0.100	1.283	0.206
4	2.607	0.050	2.746	0.036	0.696	0.124	1.362	0.224
5	2.533	0.068	2.730	0.056	0.804	0.147	1.325	0.233
7	2.492	0.032	2.706	0.037	0.787	0.133	1.273	0.220
8	2.615	0.078	2.738	0.042	0.613	0.122	1.390	0.236
9	2.49	0.042	2.701	0.039	0.928	0.129	1.540	0.125
10	2.507	0.030	2.778	0.037	0.918	0.128	1.537	0.127
11	2.605	0.065	2.723	0.059	0.651	0.139	1.167	0.247
Between-person mean and variability
Day	DET		IDN		OCL		OBK
	mean	SD	mean	SD	mean	SD	mean	SD
1	2.629	0.120	2.768	0.057	0.643	0.109	1.120	0.338
25	2.607	0.095	2.753	0.068	0.737	0.131	1.314	0.254
Lim et al.	2.59	0.12	2.77	0.09	0.94	0.1	1.23	0.2

RMSSDs that exceed SD at Day 25 are presented in . RMSSDs that exceed 0.5 SD at Day 25 are presented in bold. Participant 6 was left out intentionally because this participant did not perform the cognitive tests. For the DET and IDN, the base 10 logarithmic transformation of the reaction times for correct responses was the primary outcome. For the OBK and OCL, the arcsine transformation of the proportions of correct responses was the primary outcome.

For the DET, most or all daily changes were within one between-person SD for all participants; for the IDN, some daily changes were outside one between-person SD for some participants; and for the OCL and OBK, some daily changes were outside one between-person SD for most participants (Fig. 1). Table 2 shows that the RMSSD ranged between 0.100 and 0.150 for the OCL and exceeded the between-person variability at Day 25 (SD: 0.131) for five participants, and half of the between-person variability (i.e., 0.5 SD at Day 25) for all participants. For IDN, the RMSSD ranged between 0.036 and 0.069, exceeding the between-person SD of 0.068 for one participant and half of the between-person SD for all participants. For the DET and OBK, the intra-individual variability (range: 0.030–0.088 and 0.125–0.248, respectively) exceeded half of the between-person SD (0.095 and 0.254, respectively) for six and nine people, respectively (Table 2). In sum, most participants showed variability in cognitive performance that exceeded half of the between-person differences.

DISCUSSION

Intra-individual variability in cognitive tasks was substantial in this sample of older persons with cognitive impairments and depressive symptoms, supporting previous reports [1, 2]. Therefore, clinici-ans and researchers must be cautious when interpreting the results of studies that use single assessments. For example, if we define a change of one SD as a clinically meaningful change [21, 22], which is conventionally considered to be a “large effect” (Cohen’s d = 1.00) [23], we must consider that such a change can happen on a daily basis within an individual. Because intra-individual variability in cognitive impairments is greater in persons with cognitive impairments than in healthy persons [3-6], the issue is even more relevant for studies focusing on persons with cognitive impairments.

Intra-individual variability was especially larger than between-person variability in more complex tasks, with the smallest discrepancy in simple reaction time (DET) and the largest discrepancy in the accuracy measure of the OCL task. This is inconsistent with a study that found more variability in speed-related cognitive tasks than in accuracy tasks in older people [24]. However, they examined the variability over two or three sessions within a week apart, while we evaluated day-to-day variability over 36–93 days. Our findings suggest that single assessments of more complex tasks in particular may yield erroneous inferences and poor group-to-individual generalizability. It is likely that cognitive aging manifests first in within-person fluctuations in performance on more complex tasks, as these performances are more difficult to maintain.

One way to address this individual-level variability would be to systematically incorporate measurement bursts at every assessment of cognitive performance, which enables researchers to evaluate cognitive change relative to a person’s within-person variability [25]. Because the magnitude of the short-term variability can be a reflection of adaptive and nonadaptive cognitive processes [26] and neural compensation [27], it may be worthwhile to evaluate its potential as an additional outcome parameter or diagnostic marker.

A second way to address this intra-individual variability would be to use a single-subject study design. This will overcome the problem of not being able to extrapolate from between-person associations toward individual persons [9-12]. Single-subject studies using time series analysis can be used to make inferences about individual persons. The generalizability of single-subject study results can be increased by performing multiple single-subject studies in individuals of the same population (direct replication) or in different settings or populations (systematic replication) [28].

Methodological considerations

This study comprised unique data from 10 older persons with cognitive impairments who completed daily assessments, enabling us to study intra-indi-vidual variability in cognitive performance in the presence of cognitive impairments. Because there were only a few missing data points, our data represent a good estimate of day-to-day variability in cognitive performance. There are some possible limitations to consider. First, our sample was not consecutively screened, and the sample size was small. Therefore, the results are likely not generalizable to a population of older people with depressive symptoms and cognitive impairments in general. Second, intra-individual variability might have been inflated due to the uncontrolled situation, as participants performed the tasks unsupervised at their homes. However, the participants received clear instructions to perform the tasks with minimal distraction in a quiet room in the house. Third, our sample was heterogeneous because of our broad inclusion criteria for MCI and the inclusion of participants with depressive symptoms. This might have inflated the within-person variability in our sample because some forms of MCI are known to be associated with fluctuations of the cognitive performance. In most participants, the etiology of MCI appeared to be related to depression. This may have inflated the within-person variability, as affective states may influence cognitive performance [29]. However, elsewhere, we showed that intra-individual changes in the affective state were associated with intra-individual changes in cognitive performance in only 2 participants [30]. Furthermore, sample heterogeneity may also affect the between-person SD. Therefore, we also included the between-person SD by Lim and colleagues [20] as our point of reference (rather than the between-person SD at Day 25), which mainly altered the results for the IDN, reducing the magnitude of its intra-individual variability below 0.5 between-person SD for 6 participants. For the DET, this was the case only for one participant. For the OCL and OBK, this was the case for none of the participants. A fourth methodological consideration was that substantial ceiling effects for the OBK occurred in two persons (Participants 9 and 10). This may have reduced the intra-individual variability in working memory in these two participants.

CONCLUSION

This study found substantial intra-individual variability in cognitive performance in older people with cognitive impairments. Therefore, the interpretation of findings based on single assessments of cognitive performance and group-aggregated results must be taken with caution in studies on cognitive impairments and decline, especially regarding complex tasks. Systematically incorporating measurement bursts of cognitive performance or using single-subject study designs are potential solutions.