Reliability and validity of the severe impairment battery (SIB) in Korean dementia patients.

This study was conducted to examine the reliability, validity and clinical utility of the Severe Impairment Battery (SIB) for a Korean population. 69 dementia patients with Clinical Dementia Rating (CDR) stages 2 or 3 were participated in this study. The SIB, Korean version-Mini Mental State Examination (K-MMSE), CDR, and Seoul-Activities of Daily Living (S-ADL) were administered. The validity of the SIB was confirmed by evaluating the correlation coefficients between the SIB and K-MMSE, CDR, S-ADL, which were found to be significant. Cronbach's alpha for the total SIB score and each subscale score showed high significance, and the item-total correlation for each subscale was also acceptable. The test-retest correlation for the total SIB score and subscale scores were significant, except for the praxis and orienting to name. The total SIB score and subscale scores were examined according to CDR. The results suggest that the SIB can differentiate the poor performances of severely impaired dementia patients. On the basis of the receiver operating characteristic (ROC), it can be concluded that the SIB is able to accurately discriminate between CDR 2 and 3 patients. The results of this study suggest that the SIB is a reliable and valid instrument for evaluating severe dementia patients in Korean population.

INTRODUCTION

Dementia is one of the most severe pathologies in old age, and as it progresses, it eventually leads to severe impairment of cognitive function, and activities of daily living, as well as behavioral problems. The progression of dementia results in global and severe impairment across all spheres of cognitive function. Therefore, it is important to evaluate the changes, which take place over time and the presence and extent of the preserved abilities as dementia progresses. This information may help in the management of severely demented patient, and could be used in the design of psychosocial and environmental interventions. The therapeutic efficacy can be also evaluated in severely demented patients if they are tested with more appropriate instruments.

Many neuropsychological instruments have been developed which are designed to assess mild to moderate cognitive impairment and are in common usage. However, as dementia progresses to the advanced stages, the ability of these conventional neuropsychological and mental status assessments to measure cognitive functions becomes increasingly limited, because many dementia patients perform at floor levels (1). Moreover, since it can be difficult to test severely demented patients, due either to their refusal or the severity of deterioration, most of the instruments commonly used in cases of mild dementia show a limited range of scores and exhibit the floor effect in the advanced stages.

One approach to the assessment of the cognitive functions of severely impaired dementia patients involves the use of observer-based rating scales. For example, the Clinical Dementia Rating Scale (CDR) (2) and Global Deterioration Scale (GDS) (3) enable the clinician to evaluate the severity of dementia and the presence or absence of symptoms on the basis of clinical interviews with family members or other informants. Another way to avoid the floor effect is to use neurologic procedures such as the Glasgow Coma Scale (4), which evaluates the presence of neurologic signs and symptoms rated in conjunction with basic cognitive and functional skills. Although these approaches adequately assess the global dementia severity, they do not provide a performance-based evaluation of the dementia patient's cognitive abilities and cannot detect relatively spared abilities in various cognitive domains. Therefore several instruments have been developed to overcome these limitations and to assess patients with severe dementia who are unable to complete standard neuropsychological tests, viz. the Severe Cognitive Impairment Profile (SCIP) (5), the Preliminary Neuropsychological Battery (BNP) (6), the Test for Severe Impairment (TSI) (7), and the Severe Impairment Battery (SIB) (8).

The SIB is a particularly reliable, valid and useful instrument for evaluating cognitive changes in dementia patients whose level of functioning is in the moderate to severe range (1). The SIB was also more apt to identify differences in the performances in the 5-10 score region of the Mini Mental State Examination (MMSE) (9), thus avoiding the floor effect (10). The information provided by the SIB appeared to be fairly independent of that obtained from the CDR and GDS in the moderate to severe stages, although the utility of the SIB in milder dementia was limited (1).

At present, the number of instruments that can be used in this country to evaluate patients with severe dementia who cannot complete conventional neuropsychological tests is very limited. Recently, Kim et al. developed the Severe Dementia Rating Scale (SDS) (11), which has a similar form to the Korean version-Mini Mental State Examination (K-MMSE) (12). The SDS is composed 30 items that are independent of the education of the patients, and allows the various cognitive domains to be assessed within a shorter period of time. However, it has two main limitations, namely that it provides limited comprehensive information about the patient's cognitive functions and that the test-retest reliability interval is very short.

This study was conducted to examine the reliability and validity of the SIB for a Korean population with the purpose of making a more comprehensive and appropriate instrument available for the evaluation of dementia patients with severe impairment. Also, the receiver operating characteristic (ROC) curve was used to determine the degree to which the SIB discriminates dementia severity and to examine its clinical utility.

MATERIALS AND METHODS

Subjects

This study included 69 (male=19, female=50) patients with the diagnosis of dementia according to DSM-IV (13) and NINCDS/ADRDA (14) criteria treated at a Geropsychiatry Clinic, Department of Psychiatry, Samsung Medical Center. 54 had Alzheimer's disease (AD), 7 had the mixed type of dementia and 8 had other types of dementia, e.g., vascular dementia, dementia due to Parkinson's disease, dementia not otherwise specified. Only patients with CDR stages 2 or 3 and scores of less than 15 on the K-MMSE were included in this study. 25 subjects were rated as CDR stage 2, and 44 as stage 3. The mean K-MMSE total score was 5.49 (SD=3.91). The mean age of the subjects was 74.4 yr (SD=10.0), and their mean length of education was 7.91 yr (SD=5.78).

Severe Impairment Battery (SIB)

Saxton et al. developed the SIB to assess patients with severe dementia who cannot complete conventional neuropsychological tests. The SIB contains 51 questions which take a total of about 20 min to administer, and the possible scores range from 0 to 100. The SIB is divided into 9 subscales, viz. social interaction skills (score 0-6), memory (score 0-14), orientation (score 0-6), language (score 0-46), attention (score 0-6), praxis (score 0-8), visuospatial ability (score 0-8), construction (score 0-4) and orienting to name (score 0-2), each of which yields individual scores. There is no cut-off score for normal subjects as the test is only intended to be used with patients known to be severely impaired. However, it is possible to grade the severity of impairment by rating those who score less than 63 on the SIB (corresponding approximately to less than 4 on the MMSE). Also, in terms of its administration, the SIB is composed of very simple one-step commands, which are presented together with gesture cues and can be repeated several times to facilitate comprehension.

The SIB used in this study was translated into Korean by three clinical psychologists, who made sure that all the characteristics and structure of the original SIB were maintained in the Korean version. Two items in the language subscale were modified. First, we changed the item 'recite the months of the year' to 'recite the days of the week', because the original item was too easy for Koreans. Second, the phrase 'people spend money' was not natural when it was translated into Korean, thus we changed it to 'I bought something'.

In terms of its administration, the original materials (photograph of spoon and cup, blocks) of the SIB were familiar to the patients. However, the patients mistook the color of certain blocks. For example, as sometimes occurs in normal elderly, those patients with severe cognitive impairment or a low educational level sometimes mistook a 'blue' block for a 'green' one. Thus, when they failed to indicate the correct color for the blue block, we informed them of their mistake before passing on to the next item. All of the items of the SIB are presented in the appendix.

Korean version-Mini Mental State Examination (K-MMSE)

The MMSE was developed to evaluate the subject's orientation, memory, attention ＆ calculation, and visuospatial and language abilities. Kang et al. conducted a study to examine the validity of newly constructed K-MMSE. It contains 30 questions, and the possible scores range from 0 to 30.

Seoul-Activities of Daily Living (S-ADL)

The S-ADL (15) was developed to assess basic activities of daily living, including self-care/hygiene, toileting, ambulation, and so on. It is composed of 12 items, and the possible scores range from 0 to 24. Using the S-ADL, the clinician evaluates the subject's functional disability by conducting interviews with the caregivers.

Clinical Dementia Rating (CDR)

The CDR assesses the cognitive performance in six categories; memory, orientation, judgment and problem solving, community affairs, home and hobbies, and personal care. The information needed to evaluate each category is obtained by semi-structured interviews with both the patient and a reliable informant. The CDR provides descriptors for each category and a global CDR score of five levels of impairment (0=no dementia, 0.5=questionable, 1=mild, 2=moderate, 3=severe).

Data analysis

Statistical analyses were performed with SPSS 11.0 for WINDOWS. The Mann-Whitney U test and Spearman correlation were used to test for significant differences in the total SIB, K-MMSE and S-ADL scores according to sex, age and education. The validity of the SIB was determined by calculating the Spearman correlation coefficients between the SIB and the K-MMSE, CDR, and S-ADL. Cronbach's coefficient alpha and item-total correlation coefficients were generated to examine the internal consistency of the SIB. Also, Spearman correlation coefficients were generated to evaluate the 3 months test-retest reliability. The Mann-Whitney U test was used to assess the differences in the mean SIB total score for the various CDR stages. Finally, a ROC curve was used to compare the sensitivity, specificity and probability of correctly discriminating dementia severity in the case of the SIB, K-MMSE and S-ADL.

RESULTS

Demographical characteristics

Sex and education had no significant effect on the K-MMSE, SIB, S-ADL or CDR, but significant correlations were found between age and the total K-MMSE score (ρ(69)=0.26, p<0.05) and age and the total SIB score (ρ(69)=0.27, p<0.05).

Validity

The construct validity of the SIB was examined by comparing the total SIB score with the total K-MMSE, CDR and S-ADL scores (Table 1). The Spearman correlation coefficient between the SIB and the K-MMSE was 0.875 (p<0.01); between the SIB and the CDR was -0.678 (p<0.01); and between the SIB and the S-ADL was -0.661 (p<0.01).

Table 1

Correlations between the SIB and other variables

SIB, Severe Impairment Battery; K-MMSE, Korean version-Mini Mental State Examination; CDR, Clinical Dementia Rating; S-ADL, Seoul-Activities of Daily Living.

*p<0.01.

Reliability

The Cronbach's coefficient alpha and item-total correlation coefficients were generated to examine the internal consistency of the SIB (Table 2). The internal consistency of the total SIB score was 0.97, and that of the SIB subscales ranged from 0.57 (orientation) to 0.94 (language). The item-total correlation for the SIB subscales was also acceptable (p<0.01).

Table 2

Internal consistency of the SIB

SIB, Severe Impairment Battery.

Data from those 15 subjects who were tested again after an interval of 3 months allowed the assessment of the test-retest reliability using Spearman correlation coefficients (Table 3). The test-retest correlation for the total SIB score was statistically significant. Significant subscale correlations were observed for social interaction, memory, orientation, language, attention, visuospatial ability and construction. However, the test-retest correlation for the praxis and the orienting to name subscales did not reach significance.

Table 3

Test-retest reliability of the SIB (n=15)

SIB, Severe Impairment Battery.

*p<0.05, †p<0.01.

SIB score according to dementia severity

The subjects were separated into two groups according to their dementia severity using the CDR with the aim of measuring the total SIB score and subscale scores (Table 4). As a result, the difference in the total SIB score and its subscale scores between the CDR 2 and CDR 3 groups was significant, except for the orienting to name subscale. Despite their pervasive deficits, the more severely demented patients belonging to the CDR 3 group showed a wide range of total SIB score and subscale scores.

Table 4

Means and standard deviations of the SIB for each CDR group

SIB, Severe Impairment Battery; CDR, Clinical Dementia Rating.

Total scores of the SIB subscale: Social interaction (6), Memory (14), Orientation (6), Language (46), Attention (6), Praxis (8), Visuospatial (8), Construction (4), Orienting to name (2).

In the analysis of impairment in the individual cognitive domains, the CDR 2 patients had greater impairment in the individual cognitive domains, especially in memory and orientation, as compared with that in the full score of each subscale. On the other hand, their construction, attention and social interaction abilities were comparatively maintained. The CDR 3 patients had more severe impairment than the CDR 2 patients in all cognitive domains, and this greater impairment was especially notable for construction, attention and praxis.

Receiver Operating Characteristic (ROC) Curve

The ROC curve was used to determine the degree to which the SIB allows the dementia severity to be discriminated (CDR 2 vs. CDR 3). The sensitivity and specificity levels of the SIB were also compared with those of the K-MMSE and S-ADL (Table 5). The sensitivity and specificity of the SIB in the differentiation of the CDR 2 and CDR 3 patients were both 88% when the cut-off score was 62.5. When the cutoff score of the K-MMSE was 5.5, the corresponding sensitivity and specificity were 86% and 88%, respectively. The sensitivity and specificity of the S-ADL were 84% and 82%, respectively, when the cut-off score was 6.5. Grossly, the sensitivity and specificity of the SIB were higher than those of both the K-MMSE and S-ADL.

Table 5

Sensitivity and specificity of the SIB, K-MMSE and S-ADL

SIB, Severe Impairment Battery; K-MMSE, Korean version-Mini Mental State Examination; S-ADL, Seoul-Activities of Daily Living; AUC, Area Under Curve.

The Area Under the Curve (AUC) for each test revealed that the SIB (AUC=90.6%) was more accurate than both the K-MMSE (AUC=90%) and S-ADL (AUC=88.1%) in the differentiation of the CDR 2 and CDR 3 patients. The diagnostic accuracy of the K-MMSE was similar to that of the SIB. However, as a result of the diagnostic accuracy of the K-MMSE subscales, it was found that only the place orientation, registration, language and visuospatial subscales could be usefully discriminated between the CDR 2 and CDR 3 patients using this instrument (Table 6). The diagnostic accuracy of the S-ADL was lower than that of both the SIB and K-MMSE. Fig. 1 shows the results in the form of the ROC curve.

Table 6

AUC of the K-MMSE subtests

AUC, Area Under Curve.

*p<0.01.

Fig. 1

ROC curves of the SIB and K-MMSE.

DISCUSSION

In this study, the reliability and validity of the SIB were examined, and its clinical utility for discriminating dementia severity was proved using the ROC curve. The results of this study suggest that the SIB is a reliable, valid, and useful instrument for evaluating severely impaired dementia patients.

The significant correlation that was obtained between the SIB and the K-MMSE, CDR and ADL suggests that the SIB has appropriate construct validity, as was previously observed. In particular, the high correlation between the total SIB and K-MMSE scores indicates that the SIB evaluates global cognitive functioning in patients with severe dementia. Cronbach's coefficient alpha for the total SIB score and each subscale score and the item-total correlation for the SIB subscales showed high significance, indicating that the SIB has excellent internal consistency. The test-retest correlation for the total SIB score and each subscale scores was relatively significant. This result suggests that the performance on this instrument has acceptable stability over time. However, the test-retest correlation for the praxis subscale was not significant in this study, whereas the one month test-retest correlation for this subscale was 0.63 (p<0.01) in a previous study (1). This discrepancy may be attributed to the change in the functioning of the patients caused by the long retest interval. The test-retest correlation for the orienting to name subscale was not significant either, possibly due to the fact that the difference in the orienting to name subscale score was not notable, because almost all of the patients were successful in this subscale.

The total SIB score and the SIB subscale scores were compared according to the dementia severity using the CDR. It was found that the difference in the total SIB score and its subscale scores between the CDR 2 and CDR 3 patients was significant, except for the orienting to name subscale. Despite their pervasive deficits, the more severely demented patients belonging to CDR 3 showed a wide variation in both the total SIB scores and the scores on each subscale. These results suggest that the SIB is useful for evaluating dementia patients with a moderate to severe range of functioning, and can differentiate between the poor performances of very severely impaired dementia patients. The SIB can also avoid the floor effect that was frequently observed in conventional neuropsychological tests used for evaluating patients in the advanced stages. In the analysis of the impairment in individual cognitive domains according to dementia severity, the CDR 2 patients had greater impairment in memory and orientation than other cognitive domains. On the other hand, the construction, attention and social interaction abilities were comparatively maintained. The CDR 3 patients showed more severe impairment than the CDR 2 patients in all cognitive functions. In particular, greater impairment was observed in the construction, attention and praxis subscales. However, the difference in the orienting to name subscale score between the CDR 2 and CDR 3 patients was not significant. This result indicates that recognition of self is relatively maintained until the advanced stages of dementia. These aspects of the comparative decline in various cognitive domains are consistent with the clinical course of AD, so the result of this study should provide us with a better understanding of natural history in AD. However, there is a limitation in that this study included not only AD patients but also patients with other forms of dementia. Therefore, it is necessary to compare the different aspects of cognitive impairment by dementia subtype in a further study.

On the basis of the ROC curve, it can be concluded that the SIB was very useful for discriminating between CDR 2 and CDR 3 patients, as was the total K-MMSE score. However, the time orientation, attention/calculation and recall subscale scores were at floor and consequently, these subscales were not apt to discriminate dementia severity in the moderate to severe stages of dementia. The S-ADL had a somewhat lower sensitivity, specificity and diagnostic accuracy than those of the SIB and K-MMSE. These results indicate that the SIB has greater clinical utility than the K-MMSE and S-ADL, because it can provide comprehensive information about the changes in various types of cognitive functioning (i.e. social functioning, memory, language, and so on) of the patients and can discriminate between the relative impairment in the cognitive domains in the final stages of the disease.

This study has several limitations. First, the dementia subtype was not taken into consideration. Because those cognitive functions that are more impaired or spared may differ according to dementia subtype, it is important that the different aspects of cognitive impairment be examined by dementia subtype in a further study. Second, there is a possibility that the change in the functioning of the patients caused by the long retest interval may have affected the test-retest correlation coefficient.