A systematic review of the assessment tools used to measure metamemory in patients with brain injury.

[Purpose] The purpose of this study was to provide various assessment tools for metamemory of a patient with brain injury to develop accurate metamemory measurement methods which can be effectively applied in rehabilitation training programs in clinics.
[Subjects and Methods] For this study, we carried out a systematic document search of the literature for articles published between January, 2001, and December, 2013, in order to review studies of measurement tools used to assess the metamemory of patients with brain injury.
[Results] The initial search of the databases yielded 57 articles, 48 of which did not meet the inclusion criteria. Thus 9 studies were included in our review. The quality level of all the studies was IIA2b and PEDro scale scores were found to range from 4 to 6.
[Conclusion] More studies need to be made, using various measurement tools for metamemory of more diverse patient groups. We believe studies should be made of the assessment of metamemory and interventions for it in accordance with the diagnoses and the cognitive levels of the patients with brain injury.

INTRODUCTION

A brain injury is any injury occurring in the brain of a living organism. Brain injuries can be classified in several ways. Primary and secondary brain injury are terms used to classify the injury processes that occur in brain injury, while focal and diffuse brain injury are terms used to classify the extent or location of injury in the brain. Specific forms of brain injury include: brain damage, the destruction or degeneration of brain cells traumatic brain injury, damage that occurs when an outside force traumatically injures the brain stroke, a vascular event causing damage in the brain and acquired brain injury, damage to the brain that occurs after birth, regardless of whether it is traumatic or nontraumatic, or due to an outside or internal cause1).

Brain injury in adult occurs either from a sudden onset, such as external injury, stroke, brain tumor, hypoxia, brain infection, a metabolic disorder, dementia, or a psychiatric disease (including schizophrenia and depression), or from an insidious onset, such as a neurological disease, including addiction to alcohol or drugs and aging2). Although physical disabilities are the most common results of brain injury, difficulties with cognitive functions are actually the main obstacles preventing patients’ return to society3, 4).

Metamemory is the knowledge of memory, as well as knowledge related to storing and drawing out awareness and information, and its concept was first defined by Flavell5). Flavell6) added the theory of mind to metamemory to generalize the concept of metacognition, focusing on the diverse aspects of people’s cognition7). That is, metacognition is the adjustment and the control of knowledge in cognition processes, such as memory, understanding, concentration and problem solving, and of cognition activities. Metamemory is a concept focused on the memory, and among the cognition processes, is a type of metacognition7,8,9,10,11).

Memory is an important element for all the tasks of cognition, and everything is related to it, from a simple task such as remembering a person’s name to difficult tasks, such as understanding and using a language, and setting up an objective12). That is, memory is both a process to acquire new information, keep it, and recall it when necessary, and a means of recalling the information stored in the long-term memory13), as well as enabling analysis of the surrounding environment and decisions14, 15).

Memory training programs have recently focused on changing the methods of paying attention to memory, rather than merely providing opportunities for memory training16). In previous studies, Youn, Lee, Kim & Ryu17) conducted a multi-strategy memory training program, onto which the concept of metamemory was grafted, with an experimental group of 16 of 32 healthy elderly subjects and compared the results. The experimental subjects showed remarkably improved memory performances, and also showed improvement in other cognition areas, for which they did not receive training. A memory strategy which is usable for effective memory performance needs to be chosen to retain knowledge, and to know the methods to practically use it18). In the rehabilitation of a patient with brain injury who is expected to have cognitive impairment, metamemory is an important element which influences not only the cognitive function area but also the motor function and the daily life performances. However, too few studies have been made of the metamemory of patients with brain injury.

There have been many studies of the metamemory of healthy aged people, but little study has been made of the measurement and the effects of metamemory for brain injured patients. In most studies of healthy aged people, metamemory was measured using the metamemory in Adulthood (MIA)19) questionnaire, and their results have revealed that health, depression and self-efficacy influence metamemory. MIA is a self-assessment tool of memory knowledge and memory, which consists of 108 questions scored on a five-point scale. As it has a large number of questions, and certain cognitive functions are needed to understand the contents of the questionnaire, it is difficult to use it to assess brain-injured patients. Therefore, in this research, we systematically review study results of metamemory assessments of brain-injured patients, to try to find which metamemory tools can be applied most effectively in accordance with the cognitive levels of brain injured patients.

In this study, we analyzed the assessment tools of metamemory which were used to assess patients with brain injury from 2001 to 2013, in order to collect information on diverse assessment tools for metamemory which can be used to assess patients whose cognitive functions have been lowered by brain injury. The purpose of this study was to identify various assessment tools of metamemory for patients with brain injury according to cognitive level, to develop accurate metamemory measurements for brain injury patients which can be effectively used in rehabilitation training programs in clinics.

SUBJECTS AND METHODS

For the study, we carried out a systematic document search of the literature for articles published between January, 2001, and December, 2013, in order to review a variety of studies on the measurement tools used to assess metamemory of patients with brain. The selection and exclusion criteria were as follows. The selection criteria were studies of human beings, studies for which we could retrieve complete documents, studies of subjects who were patients diagnosed with brain injury, and studies which described the measurement tools which were used to assess metamemory. The exclusion criteria were studies of animals, studies in which the subjects were patients not diagnosed with brain injury, studies in which the assessments were not used to measure metamemory, and books, master’s or doctor’s theses, comments, or conference or presentation papers.

From January 2001 to December 2013, a systematic review of the scientific literature in the databases of PubMed Central, COCHRANE and MEDLINE was carried out. The following search strategy was used: “brain injury” OR “brain injured” AND metamemory AND assessment OR “brain injury” OR “brain injured” AND metamemory AND evaluation OR “brain injury” OR “brain injured” AND metamemory AND test. As a result of this search, a total of 57 articles were retrieved. After analysis, 9 articles fulfilling the inclusion criteria were chosen for further analysis.

The selected studies were qualitatively evaluated by two researchers independently based on the level of evidence criteria used by the American Occupational Therapy Association (AOTA). The level of evidence suggested by the AOTA is shown in Table 1. It formulates objective levels for each study based on the type of study design, sample size, internal validity and external validity20). In this study, two researchers individually reviewed the level of evidence and, in cases of disagreement, a consensus was reached through discussions between the researchers and recorded.

Table 1.

Levels of evidence for occupational therapy outcomes research

	Definition
Level of evidence for design
I	Randomized control trial
II	Non-randomized control trial—two groups
III	Non-randomized control trial—one group (one treatment) pretest and posttest
IV	Single-subject design
NA	Narratives; case studies
Level of evidence for sample size
A	n ≥ 20 per condition
B	n < 20 per condition
Level of evidence for internal validity
1	High internal validity: no alternate explanation for outcome
2	Moderate internal validity: attempt to control for lack of randomization
3	Low internal validity: two or more serious alternative explanations for outcome
Level of evidence for external validity
a	High external validity: Participants represent population and treatments represent current practice
b	Moderate external validity: Between high and low
c	Low external validity: Heterogeneous sample without being able to understand whether effects were similar for all diagnoses or treatment does not represent current practice.

The clinical trials were analyzed for methodological quality using the PEDro Scale21), which has 10 items for the evaluation of internal validity and statistical analysis of randomized controlled trials. Each adequately fulfilled item receives 1 point and contributes to the score which has a maximum 10 points. The PEDro scale contains 10 items: “random allocation,” “concealed allocation,” “baseline comparability,” “blind subjects,” “blind therapists,” “blind assessors,” “adequate follow-up,” “intention-to-treat analysis,” “between-group comparisons,” and ”point estimates and variability.”

RESULTS

The initial search of the databases yielded 57 articles, 48 of which did not meet the inclusion criteria. Thus 9 studies were included in our review.

Table 2 shows the methodological quality analysis results of the studies according to the PEDro Scale. The quality level of all the studies was IIA2b and the PEDro scale scores ranged from 4 to 6, indicating that all the studies were proper in the aspect of their methodologies. Table 3 shows the characteristics of the samples, metatmemory measurement instruments and the main outcomes and methodological characteristics of the studies.

Table 2.

Methodological quality analysis results according to the PEDro scale

Author	PEDro scale
	1	2	3	4	5	6	7	8	9	10	S
Eslinger et al.28)		*					*	*	*	*	5
Maureen & Jonathan29)		*	*				*	*	*	*	6
Stephanie et al.33)		*					*	*	*	*	5
Sarah & Sandra30)		*					*	*	*	*	5
Jonathn & Maureen31)		*					*	*	*	*	5
Trevor et al.35)		*						*	*	*	4
Stephanie et al.34)		*					*	*	*	*	5
Michele et al.36)		*					*	*	*	*	5
Lauren et al.32)		*					*	*	*	*	5

1. Random allocation, 2. Allocation concealment, 3. Similar prognostic at the beginning. 4. Subject concealment, 5. Blinding of therapist, 6. Blinding of therapists, 6. Blinding of reviewer, 7. Follow up over 85% of significant result and report of the punctual assessment, 8. The analysis includes an intention-to-treat analysis, 9. Result from the statistical comparisons among groups are shown at least for one of the measurements, 10. Variability measurement of, at least, a significant result, S=Score, *Criteria fulfilled

Table 3.

Features of the retrieved metatmemory measure instruments, and the main outcomes

Author	Sample	Metamemory measurement instruments	Main outcomes	Level of evidence	PEDro scale score
Eslinger et al.28)	-Total n=49-FTD (n=27)-AD (n=11)-Healthy controls (n=11)	- Self awareness- Self knowledge	-The FTD sample as a whole showed significantly less behavioural self-awareness and self-knowledge than the AD and healthy control samples (p<0.05).	IIA2b	5
Maureen & Jonathan29)	- Total n=42-moderate-to-severe CHI (n=21)-neurologically normal control (n=21)	- Confidence rating- FOK rating	-CHI participants demonstrated less accurate recall but accurate ability to judge their recall performance.-CHI participants demonstrated intact feeling-of-knowing judgments when providing binary judgments, but demonstrated difficulties making finer discriminations on an ordinal scale.	IIA2b	6
Stephanie et al.33)	-Total n=44-AD (n=24)-Healthy elderly volunteers (n=20)	-Metamemory battery: GK, VJOL, NJOL.	-Patients who were rated as aware of their memory loss remained well calibrated over the course of the task whereas those rated as relatively unaware grew over-confident in their predictions F (1, 33) = 4.19, p = 0.02.	IIA2b	5
Sarah & Sandra30)	-Total n=55-PPA (n=14),-bvFTD (n=11)-PrAD (n=15)-Older NC (n=15)	- SAS- SMS	-PPA and normal control groups performed very similarly on control (Weight and Eyesight) and cognitive domains, whereas bvFTD and PrAD patients were unable to accurately assess Memory.	IIA2b	5
Jonathn & Maureen31)	-Total n=58-Moderate to severe TBI (n=29)-Neurologically normal controls (n=29)	-Memory self-awareness-Memory self-monitoring	-No significant group differences emerged in memory self-awareness or memory self-monitoring.	IIA2b	5
Trevor et al.35)	-Total n=687-With dementia (n=152)-Without dementia (n=535)	-A brief metacognition questionnaires	-Among individuals without dementia, metacognition significantly predicted 3MS change (p=0.027) and IQCODE ratings (OR=4.0, 95% CI= 1.2 − 13.8 spatient=0.029).-For those with dementia, there was a weak, inverse relationship between 3MS change and metacognition (r = −0.16, p=0.056). IQCODE ratings were not significantly associated with metacognition (p=0.729).	IIA2b	4
Stephanie et al.34)	-Total n=92-Mild AD (n=42)- Healthy elders(n=50)	- Metamemory battery: GK, VJOL, NJOL.	-45% of AD subjects were classified as aware (AAD) and 55% as unaware (UAD) based on clinical ratings and supported by metamemory testing (p = 0.015).-Capacity was impaired in each of the AD groups as compared to the healthy elders F (2, 67) = 17.63, UAD, p < 0.01; AAD, p = 0.01).	IIA2b	5
Michele et al.36)	-Total n=24-Cognitively healthy (n=12)-People with memory impairment due to MCI or early AD (n=12)	- MARS- MFS of the MARS	-Control participant memory self-appraisal ratings were significantly positively correlated with their study partners’ ratings (rs = −0.60, p = 0.04). -In contrast, self-appraisals of memory by participants with diagnosed memory disorders did not correlate with their study partners’ ratings (p > 0.05).	IIA2b	5
Lauren et al.32)	-Total n=122-bvFTD (n=49)-AD (n=73)	-Patient’s self- appraisal of neuropsychological test (BNT, TROG, Rey Complex Figure Immediate Free Recall) performance.	-bvFTD patients were poor at evaluating their own performance in all cognitive tests, with no significant correlations between self-appraisal and actual performance.-By contrast, poor self-appraisal by AD patients was restricted to episodic memory performance.	IIA2b	5

FTD: Frontotemporal dementia. AD: Alzheimer’s disease. CHI: closed-head injury. PPA: Primary progressive aphasia. bvFTD: Behavioral variant frontotemporal dementia. PrAD: Probable Alzheimer’s disease. NC: normal controls. FOK: Feeling-of-Knowing. GK: General Knowledge. VJOL: Verbal Episodic Judgment of Learning. NJOL: Nonverbal Episodic Judgment of Learning. SAS: Self-awareness scores. SMS: Self-monitoring scores. MARS: Memory Awareness Rating Scale. MFS: Memory Function Scale. BNT: Boston Naming Test. TROG: Test of Receptive Grammar. IQCODE: Informant Questionnaire of Cognitive Decline in the Elderly.

The studies involved a total of 1,173 subjects, comprising of 639 (59.08%) healthy subjects, 239 (20.38%) dementia patients, 177 (15.09%) Alzheimer’s disease patients, 29 (2.47%) moderate to severe traumatic brain injury (TBI) patients, 21 (1.79%) moderate to severe closed-head injury patients, 14 (1.19%) primary progressive aphasia patients (Table 4).

Table 4.

Classification of the subjects and their frequencies

Subject classification	N (%)
Neurologically normal control	693 (59.08)
Dementia	239 (20.38)
Alzheimer’s disease (AD): include FTD, bvFTD	177 (15.09)
Moderate to severe TBI	29 (2.47)
Moderate-to-severe closed-head injury (CHI)	21 (1.79)
Primary progressive aphasia (PPA)	14 (1.19)
Total	1,173 (100)

AD: Alzheimer’s disease. FTD: Frontotemporal dementia. bvFTD: Behavioral variant frontotemporal dementia. TBI: Traumatic brain injury. CHI: closed-head injury. PPA: Primary progressive aphasia.

When we classified the studies in accordance with the characteristics of the tools used to measure metamemory, self-appraisal of a neuropsychological test or task performance was performed in five (55.56%) of the studies and the ratings were indicated by making a mark along a visual analogue scale. The metamemory battery was used in two of the studies, while the Metacognition Ratings and the Memory Awareness Rating Sacle (MARS) were used respectively in the remaining two studies (Table 5).

Table 5.

Classification of the assessment tools used to measure metamemory and their frequency of use

Metamemory measurement instruments	N (%)
Participant’s self-appraisal of neuropsychological test or task performance	5 (55.56)
Metamemory Battery	2 (22.22)
Metacognition Ratings	1 (11.11)
Memory Awareness Rating Sacle (MARS)	1 (11.11)
Total	9 (100)

DISCUSSION

Metamemory may largely be divided into descriptive metamemory and procedural metamemory. Descriptive metamemory might be regarded as knowledge about what is required in each memory task, about the limited capacity of the memory system, related with such requirements, and about the alternative strategies to overcome those limits22). Procedural metamemory is ‘the inspection and control of the memory process’, where as inspection means the inspection of the extent of performance, performed while carrying out a memory task, to understand the current position relative to that of the objective, and control is practical action for memorization of things, such as planning, instruction, distribution of efforts, attention and assessment.

The methods to measure metamemory are diverse. According to Cavanaugh and Perlmutter23) descriptive metamemory may be measured independently of the assessment of the memory activities, while procedural metamemory should be measured together with the assessment of the memory activities. Descriptive metamemory generally measures the information or the facts about the memory the subject keeps, that is, the knowledge about the memory capacity, tasks and strategies, as well as about their mutual reactions. Procedural metamemory, on the other hand, measures on the activities to awaken and control the inspection of ongoing memory.

The relationship between cognition and motor control after neurological damage due to factors such as stroke has significant implications for understanding the recovery of motor function24). Cognitive-motor interference refers to the phenomenon that occurs when 1 or 2 tasks that interfere with each other are being performed, such as engaging in cognitive and motor tasks simultaneously25). Interference between cognitive tasks and motor control activities (such as walking) is important for the functional improvement of patients with neurological deficits24). Thus, the therapist must address both cognitive and motor training when planning rehabilitation therapy26).

The main benefit of an integrated exercise and cognitive therapy program is an increase in activity and participation, as well as an increase in neural plasticity. Improvements in physical activity and cognitive function may also help to reduce neural degenerative structures, such as beta-amyloidal plaques and tangles27). Accordingly, providing exercise programs based on cognitive programs including metamemory evaluation and intervention are effective rehabilitation training for patient with brain injury.

The classification of the metamemory measurement tools used in this study and the analysis of the frequencies of their use showed that five of the studies we reviewed used self-appraisal of a neuropsychological test or task performance. In the study by Eslinger et al.28), subjects predicted their performance on a visual analogue scale (0= very low to 100= very high) before a verbal memory measurement and then they rated their actual performance in a similar fashion. Immediately following the test, the subjects rated how well they performed by placing an X on the visual analog scale. In the remaining four studies metamemory wassimilarly measured. In the study by Maureen and Jonathan29), confidence rating and feeling-of-knowing (FOK) rating were assessed; in the study of Sarah and Sandra30); self-awareness scores (SAS) and the self-monitoring scores (SMS) were used; in the study of Jonathn and Maureen31), memory self-awareness and the memory self-monitoring were measured; and in the study of Lauren et al.32), self-appraisals of neuropsychological tests were made. Although the terms used in each study are different from those of others, we surmised that the contents being measured and the methods were similar. This indicates that they measured descriptive metamemory and procedural metamemory together in most studies, as mentioned above.

The review results of the concepts of the primary factors used in the metamemory assessment in each study are described below.

The confidence rating used in the study by Maureen and Jonathan29) was the correctness of the answer, rated on a 7-point Likert scale (1=‘I am not all sure that my answer is correct’ to 7=‘I am absolutely sure that my answer is correct’). And a Feeling-of-Knowing (FOK) rating was also made on a 7-point Likert scale (1=‘I am absolutely sure that I will not recognize the correct answer’ to 7=‘I am absolutely sure that I will recognize the correct answer’).

The self-awareness scores (SAS) used in the study by Sarah and Sandra30) were calculated on the basis of pre-test self-estimates. The self-estimate visual analog scale was converted to a percentage of the best possible score, reflecting the number of centimeters from the end of the line, in which so an estimate of ‘average’ would be (9 cm/18 cm) × 100 = 50%, comparable to an ‘average’ T score of 50, and a slightly above average estimate (e.g., 11 cm/18 cm) would be comparable to a T score of 61, etc. Self-monitoring scores (SMS) were also calculated, using a similar method.

Memory self-awareness used in the study by Jonathan and Maureen31) was assessed by the participants predicting the amount of information they would remember before completing list-learning and visual-spatial memory tasks. And Memory self-monitoring was assessed by the participants’ ability to increase the accuracy of their predictions after experience with the tests.

In the study by Lauren et al.32), self-appraisal of neuropsychological test [Boston Naming Test (BNT), Test of Receptive Grammar (TROG), Rey Complex Figure Immediate Free Recall] performance was conducted. Ratings were indicated by making a mark on a 50-mm line, ranging from very poor performance to very high performance (0=very low to 100 =very high).

The Metamemory Battery used to measure metamemory in the studies by Stephanie et al.33) and Stephanie et al.34) was composed of General Knowledge (GK), Verbal Episodic Judgment of Learning (VJOL), Nonverbal Episodic Judgment of Learning (NJOL). Throughout the metamemory tasks, participants were required to make judgments about their own performance using a three-point scale (Yes, Maybe, or No). The rating scale was printed on 8.5″ × 11″ paper, with the rating items (Yes − Maybe − No) written vertically in the center of the page, and was placed directly in front of the participant for the duration of the metamemory battery. The metamemory test, a modified episodic feeling of the Alzheimer’s disease (AD) knowing task, was designed as part of a larger ongoing study on metamemory.

In the study by Trevor et al.35), Metacognition Ratings were used. Participants completed a brief, 7-item questionnaire that was adapted from other published instruments assessing cognitive and functional status. Three of the questions were designed to assess functional changes within the past three years and four questions were designed to assess cognitive changes occurring within the same time interval.

In the study of Michele et al.36), the Memory Awareness Rating Scale (MARS) was used. The Memory Function Scale (MFS) of the MARS indexes memory awareness is the discrepancy between a study partner’s report and the subject’s self-report on parallel forms. The Memory Function Scale of the MARS was our index of memory self-appraisal accuracy.

The major results of the studies analyzed can be summarized as follows. Although the metamemory of the patients with brain injury was mostly found to be low, no significant group differences emerged in memory self-awareness or memory self-monitoring in the study of Jonathan and Maureen31). These findings suggest that in the early stages of recovery from brain injury, individuals with TBI can demonstrate accurate memory self-awareness. This result contrasts with some other studies in the literature, which suggest that persons in the early stages of recovery from TBI exhibit difficulties with memory self-awareness37,38,39,40). The techniques used in these earlier studies, mostly relied on contrasting self-reports with ratings provided by a family member or rehabilitation staff. This methodology has been referred to as “off-line assessment” because the questionnaires and interviews used to gather information about the patient’s memory beliefs and knowledge are temporally distal from any particular memory task41).

Metamemory has been examined only minimally in patients with brain injury. Although we carried out a systematic review of the metamemory assessment tools for brain injured patients, the study cases are few and their generalization is limited, making it difficult to propose the most useful metamemory assessment tools appropriate for different cognitive levels of brain-injured patients. Most studies of healthy aged people used MIA (metamemory in Adulthood)19) to let them measure their own memory knowledge and memory beliefs. MIA comprises 108 questions, and has seven sub-measures, such as strategy, task execution capability, change, achievement, control and anxiety. They are measured on five-point scales, with the score ranging from 70 to 350, with higher scores indicating higher metamemory.

This review was conducted to seek assessment tools for brain injured patients. According to our research, for brain injured patients who have lowered cognitive level, the measurement is best made by letting them carry out simple tasks, rather than through a questionnaire with many questions, or by the self-assessment measures measured before and after conducting a neurological examination. This is not an objective and standardized measurement, but it enables a qualitative assessment of metamemory, which may be used to determine significant rehabilitation intervention objectives and intervention plans: initially those that examine knowledge about the memory task, secondly, those that examine general awareness of memory capabilities, and finally, those that examine the frequency and type of forgetting in each task.

In conclusion, you can get information through the measurement of the metamemory of a patient with brain injury, and apply it to a rehabilitation program. The methods of planning, instruction, distribution of effort and attention, etc. should help the execution of a functional task, through the process of understanding the present situation relative to the objective, by proposing alternative strategies to overcome the limited knowledge, about what is required in the task, and the memory system, as well as by inspecting the extent of performance when carrying out the task.

The classification of the subjects covered by this review shows that most of them were patients with dementia or Alzheimer’s disease (AD). There was no study including stroke patients, a patient group often encountered in clinical practice. Patients with stroke may have been included in the patient group with primary progressive aphasia (PPA), but we could not confirm this. More studies need to be conducted, using various measurement tools for metamemory of more diverse patient groups. We believed studies should be made of the assessment of metamemory and interventions for it in accordance with the diagnoses and the cognitive levels of the patients with brain injury. In future systematic reviews of such diverse studies, would enable more effective rehabilitation training programs to be proposed for patients, based on accurate metamemory measurement results, in accordance with the diagnostic groupings of the patients with brain injury, as well as in accordance with their cognitive functions.