Screening for Normal Cognition, Mild Cognitive Impairment, and Dementia with the Korean Dementia Screening Questionnaire.

OBJECTIVE: The Korean Dementia Screening Questionnaire (KDSQ) is an informant-based instrument used to screen for cognitive dysfunction. However, its ability to only dichotomously discriminate between dementia and normal cognition has been previously investigated. This study investigated the ability of the KDSQ to classify not only dichotomous but also multiple stages of cognitive dysfunction.
METHODS: We examined 582 participants. Receiver operating characteristic (ROC) curves were used to determine dichotomous classification parameters. Multi-category ROC surfaces were evaluated to classify the three stages of cognitive dysfunction.
RESULTS: Dichotomous classification using the ROC curve analyses showed that the area under the curve was 0.92 for dementia for subjects without dementia and 0.96 for dementia in controls. Simultaneous multi-category classification analyses showed that the volume under the ROC surface (VUS) was 0.57 and that the derived optimal cut-off points were 2 and 8 for controls, MCI, and dementia. The estimated Youden index for the KDSQ was 0.48, and the derived optimal cut-off points were 5 and 10. The overall classification accuracy of the VUS and Youden index was 61.2% and 58.6%, respectively.
CONCLUSION: The KDSQ is useful for classifying dichotomous and multi-category stages of cognitive dysfunction.

INTRODUCTION

The global prevalence of dementia has rapidly increased, thus resulting in considerable health and social implications [1]. In Korea, the aged population has quickly grown, and the prevalence of dementia has rapidly increased as a result [2]. Dementia has thus become a feared disease in Korea. In April 2007, the Korean Ministry of Health and Welfare began a new National Screening Program for Transitional Age for people who are 66 years old [3]. In this program, all participants complete part of the Korean Dementia Screening Questionnaire (KDSQ) to screen for cognitive dysfunction.

The KDSQ is an informant-based questionnaire that addresses changes in elderly subjects’ cognitive performance over the previous year [4]. Numerous screening tools for cognitive dysfunction are already available, but the KDSQ has been widely used in Korea because of its ease of use [3-5]. The KDSQ has demonstrated high validity and reliability for screening for early dementia [4].

Although the purpose of screening is to correctly distinguish subjects with dementia from those with normal cognition, cognitive capacity does not always fall into dichotomous stages in the clinical setting. A number of disease processes usually involve pre-disease or intermittent stages as patient’s progress from health to full disease. Mild cognitive impairment (MCI) is a transitional state between normal cognition and dementia [6,7]. However, most screening tools, including the KDSQ, have been evaluated for their ability to make only dichotomous distinctions between normal cognition and dementia or between normal cognition and cognitive dysfunction [4,8,9]. In clinical settings, these dichotomous categories have not been always useful for classifying patients by graded stages of cognitive dysfunction.

METHODS

Subjects

The study subjects were those who visited a memory clinic at a university hospital in the Republic of Korea from January 2008 to October 2016 and were referred for neuropsychological testing. A total of 582 subjects were recruited with a matched samples design: 194 subjects with dementia, 194 subjects with MCI, and 194 subjects who were cognitively normal (CN). All subjects from these three diagnostic groups were age-matched on the basis of 3-year bands, gender and 5-year bands for education. Standard clinical diagnostic criteria were used to diagnose MCI [7] and dementia [10]. CN individuals did not meet the criteria for MCI or dementia. Consensus diagnosis by a geriatric physician and neuropsychologist was used to determine each individual’s clinical status on the basis of the results of a neuropsychological battery, neurological and physical exams, and interviews with an informant, who assessed the subject’s global cognitive status, functional status, mood, and behavioral status. Neither dementia subtypes nor MCI subtypes were specifically examined. The exclusion criteria included preexisting conditions that might affect subjects’ performance on cognitive measures, such as intellectual disability, drug or substance abuse, and severe psychiatric illness. All subjects who were accompanied by an informant were included. The informants were subjects’ spouses or relatives who lived in the same household and reported no psychiatric or neurological disease themselves. This study was approved by the Institutional Review Board at our institute (AS16159), which waived the requirement to obtain informed consent.

Clinical assessment

All subjects were evaluated on the basis of medical history, informant-based history, physical and neurological examination, laboratory tests, brain magnetic resonance imaging, and a neuropsychological battery. The neuropsychological battery was used with the Korean Version of the assessment packet developed by the Consortium to Establish a Registry for Alzheimer’s Disease [11].

To compare the results between the KDSQ and other neuropsychological tests, the KDSQ [4], Mini-Mental Status Examination (MMSE) [12], and Clinical Dementia Rating (CDR) [13] scale were independently completed, and clinicians were blinded to the scores.

The KDSQ consists of three subscales (i.e., global memory function, other cognitive function, and instrumental activities of daily living), including 15 items that can detect early changes in cognitive decline to diagnose dementia [4]. Each item on the KDSQ is scored from 0 to 2, with a higher score indicating poorer function and a greater frequency. The KDSQ is not influenced by age or educational level and has shown a 0.79 sensitivity and 0.80 specificity for dementia [4].

The MMSE is a widely used instrument that estimates global cognitive function and screens for dementia [12,14]. The items on the MMSE assess orientation, attention/working memory, memory, language, and visuospatial skills. The maximum score is 30 points, and a higher score indicates better cognitive functioning.

The CDR was used to determine the presence or absence of cognitive dysfunction and, if present, to stage its severity [13]. The CDR assesses a subject’s cognitive and functional performance in the areas of memory, orientation, judgment and problem solving, community affairs, home and hobbies, and personal care.

Statistical analyses

Clinical characteristics were evaluated with chi-square tests for the difference between proportions, and one-way analysis of variance (ANOVA) or Kruskal-Wallis ANOVA was used to test differences between continuous variables after performing Levene’s test for equality of variance. Bonferroni correction was used for post hoc comparisons. To determine convergent validity, the correlation between the KDSQ and other neuropsychological tests was calculated using Spearman’s correlation analysis.

The area under the curve (AUC) can be used as a measure of a test’s dichotomous screening ability and to choose the optimal cut-off point that provides the largest AUC. Screening ability was computed to distinguish between dementia and non-dementia (MCI or CN), between cognitive dysfunction (dementia or MCI) and CN, between dementia and CN, and between MCI and CN. The corresponding sensitivity, specificity, AUC, and predictive values were also calculated. Differences between the AUCs were analyzed using the Hanley and McNeil correction [15].

Receiver operating characteristic (ROC) analysis extends these methods to instances in which more than two diagnostic or screening categories are being classified; accordingly, the volume under the ROC surface (VUS), an extension of the AUC, has been proposed [16-18]. Optimal cut-off points can be selected on the basis of either the VUS or the Youden index. These cut-off points can be computed according to the ROC surface by finding the coordinate on the surface that minimizes the Euclidean distance from the point representing a perfect classification for all categories. Alternatively, the Youden index involves the selection of the pair of cut-off points that maximizes the overall classification accuracy by maximizing the equation (x+y+z-1)÷2, where x, y, and z are the respective probabilities of correctly classifying each of the different categories. The VUS and Youden index range between 0 and 1, with 1 representing a test with perfect diagnostic accuracy. The optimal cut-off points suggested by these two methods do not tend to be the same unless x, y, and z are equal. The VUSs were compared by using omnibus ANOVA with Wald test [17]. The classification accuracy reflects the percentage of subjects who were correctly classified (number of correct classifications/sample size) [19]. In this study, the classification accuracy indicated the percentage of correct classifications of CN, MCI, and dementia.

Analyses were performed using SPSS for Windows, version 20.0 (IBM Corp., Armonk, NY, USA) and R 3.2.4 software (The R Foundation for Statistical Computing). Statistical tests were two-tailed, and α was set at <0.05.

RESULTS

Participant characteristics are summarized according to cognitive dysfunction stage in Table 1. The sample contained 52.6% women, had an average age of 66.9±9.9 years and had an average education of 8.1±4.7 years. The average KDSQ score of the analyzed sample was 7.2±6.9, with average scores of 24.2±5.1 on the MMSE and 0.4±0.4 on the CDR. As expected, the KDSQ, MMSE, and CDR scores differed across each stage in the post hoc analyses.

Table 1.

Demographic and clinical characteristics

	Clinical diagnosis			Overall p	Post-hoc test
	CN (N=194)	MCI (N=194)	Dementia (N=194)
Age (y)	65.9±10.1	67.6±9.6	67.1±10.1	>0.05
Education (y)	8.6±4.8	7.9±4.6	7.9±4.7	>0.05
Gender (% female)	52.6	52.6	52.6	>0.05
MMSE	27.9±1.7	25.1±3.2	19.6±5.4	<0.001	CN>MCI>dementia
CDR	0.0±0.0	0.5±0.1	0.8±0.4	<0.001	CN<MCI<dementia
KDSQ	2.8±2.3	4.9±3.6	14.1±7.2	<0.001	CN>MCI>dementia

CN: cognitively normal, MCI: mild cognitive impairment, MMSE: mini-mental state examination, CDR: clinical dementia rating, KDSQ: Korean Dementia Screen Questionnaire

The correlations of the KDSQ with other neuropsychological tests were statistically significant. Specifically, the KDSQ scores were negatively correlated with the MMSE scores (r=-0.649, p<0.01) and positively correlated with the CDR scores (r=0.681, p<0.01).

ROC curves were drawn to determine the dichotomous discriminatory ability of the KDSQ (Figure 1, Table 2). The AUCs were 0.92 [95% confidence interval (CI), 0.90–0.94] for dementia vs. non-dementia, 0.81 (95% CI, 0.78–0.85) for cognitive dysfunction vs. CN, 0.96 (95% CI, 0.93–0.98) for dementia vs. CN, and 0.67 (95% CI, 0.62–0.72) for MCI vs. CN. These results showed that the KDSQ differentiated these groups well; the corresponding values for the MMSE are presented in Figure 1. When discriminating dementia from non-dementia or CN, the AUCs of the KDSQ did not significantly differ from those of the MMSE (all p>0.05). However, the AUCs of the KDSQ were significantly different from those of the MMSE in distinguishing cognitive dysfunction or MCI from CN (Figure 1). Table 2 presents the optimal cut-off points and various parameters for dichotomous screening.

Figure 1.

Receiver operating characteristic (ROC) curve for dichotomous screening. A: Dementia versus non-dementia. B: Cognitive dysfunction versus CN. C: Dementia versus CN. D: MCI versus CN. *p<0.05. CN: cognitively normal, MCI: mild cognitive impairment, AUC: area under the curve, KDSQ: Korean Dementia Screening Questionnaire, MMSE: mini-mental state examination.

Table 2.

Dichotomous screening ability of the KDSQ

Parameters	Dementia vs. non-dementia (N=582)	Cog dysf. vs. CN (N=582)	Dementia vs. CN (N=388)	MCI vs. CN (N=388)
Cut-off points	≥6	≥6	≥6	≥2
AUC	0.92	0.81	0.96	0.67
Sensitivity	0.86	0.56	0.86	0.71
Specificity	0.84	0.94	0.94	0.51
PPV	0.73	0.95	0.93	0.59
NPV	0.92	0.52	0.87	0.64
LR+	5.39	9.04	13.92	1.44
LR-	0.17	0.47	0.15	0.57

KDSQ: Korean Dementia Screening Questionnaire, Cog dysf.: cognitive dysfunction, CN: cognitively normal, MCI: mild cognitive impairment, AUC: area under the curve, PPV: positive predictive value, NPV: negative predictive value, LR: likelihood ratio

The ROC surface of the KDSQ was analyzed as a measure for discriminating between CN, MCI, and dementia (Figure 2). The estimated VUS for the KDSQ was 0.57 (95% CI, 0.53–0.62), and the Youden index for the KDSQ was estimated as 0.48 (95% CI, 0.44–0.52). The plots in Figure 3 suggested that there was a separation of the distribution of KDSQ scores between CN, MCI, and dementia, and the optimal cut-off points derived were 2 and 8 by the VUS and 5 and 10 by the Youden index.

Figure 2.

Receiver operating characteristic surfaces for multi-category classification and VUSs of the KDSQ. KDSQ: Korean Dementia Screening Questionnaire, VUS: volume under the ROC curve.

Figure 3.

Boxplots of KDSQ scores and multi-category classification cut-off points based on the VUS and Youden index. CN: cognitively normal, MCI: mild cognitive impairment, VUS: volume under the ROC curve.

To evaluate the ability of each test to classify the three stages of cognitive dysfunction, the VUSs for the KDSQ and the MMSE were compared. The VUS of the MMSE was 0.60 (95% CI, 0.55–0.65). The VUS of the KDSQ did not significantly differ from that of the MMSE (p>0.05), thus demonstrating that the KDSQ did not have lower classification ability than that of the MMSE in this study.

Table 3 presents the overall classification accuracy, expressed as the proportion of correctly classified subjects. The overall classification accuracy of the VUS and Youden index was 61.2% and 58.6%, respectively.

Table 3.

Multi-category classification accuracy of the KDSQ

Model	Cut-off points	Correctly classified (%)			Overall
		CN	MCI	Dementia
VUS	≤2, 3–7, ≥8	50.5	51.5	81.4	61.2
Youden index	≤5, 6–9, ≥10	85.1	23.2	67.5	58.6

CN: cognitively normal, MCI: mild cognitive impairment, VUS: volume under the surface

DISCUSSION

This study investigated whether the KDSQ was able to classify not only two but also three stages of cognitive dysfunction. The results showed that the KDSQ had dichotomous discriminatory ability. Moreover, the KDSQ was able to simultaneously discriminate among three stages of cognitive dysfunction: CN, MCI, and dementia.

This study identified cut-off points of 6 for the dichotomous detection of dementia from both non-dementia and CN, with a sensitivity of 86% and specificity of 84–94%. A previous study has reported the cut-off point of 6 for dichotomous detection of early dementia, with a sensitivity of 79% and specificity of 80% [5]. Therefore, this study and that previous study reported the same cut-off point and efficient screening parameters for the identification of dementia [4].

No previous studies have assessed whether the KDSQ can accurately detect MCI. Here, we demonstrate that the ability of the KDSQ to detect MCI or cognitive dysfunction including MCI is statistically lower than that of the MMSE, on the basis of AUCs. Because the KDSQ was originally designed to screen for dementia, not MCI, we presumed that the KDSQ would have different discriminatory abilities for dementia and MCI.

This study suggested cut-off points that could be used in screening and classification tools for multiple stages of cognitive dysfunction in the clinical setting. The VUS and Youden index can be used to determine cut-off points for multiple stages of disease. A diagnostic test that is useless would randomly classify individuals into one of three groups with a probability of 1/6 (approximately 0.17), whereas a perfect diagnostic test would have a probability of 1 [18]. This study showed that the KDSQ had a VUS of 0.57 and a Youden index of 0.48, both of which were higher than the random classification probability of 0.17. The two optimal cut-off points derived from the VUS and the Youden index were not the same because of the different respective probabilities for correct classification (x, y, z). This study showed that although the derived optimal cut-off points based on the VUS and the Youden index included different ranges, both points showed similar classification accuracies of approximately 60%. The classification accuracy based on the Youden index was higher for normal cognition and lower for dementia than the corresponding values based on the VUS. If dementia screening is a higher priority in clinical settings, the derived cut-off points based on the VUS could be chosen. In contrast, if CN screening is a higher priority, those based on the Youden index could be used. Thus, these two cut-off points could be applied according to the different clinical needs or circumstances.

Screening tools should be demonstrated as sufficiently valid and reliable to be used for screening or classifying. The MMSE is the most widely used cognitive test and has modest accuracy for ruling out a diagnosis of cognitive dysfunction [20]. The CDR is a structured interview conducted with a patient and a reliable informant and is a valid tool for assessing the severity of cognitive dysfunction [21]. These tools are widely used worldwide, primarily to screen for or diagnose cognitive dysfunction. In this study, the KDSQ was statistically correlated with these neuropsychological tests, and the discriminatory ability of the KDSQ did not significantly differ from that of the MMSE. We suggest that the KDSQ is useful for screening or classifying cognitive dysfunction in clinical settings, because the KDSQ was found to have similar psychometric properties to those of well-known screening tools and was able to detect and classify cognitive dysfunction.

The limitations of this study include that it applied a matched samples design and that the sample was not representative of the entire population, including only people who visited clinics. Subjects with subjective memory impairment (SMI) might have been recruited as CN; however, SMI is a known risk factor for dementia in older subjects without cognitive impairment. Neither dementia subtypes nor MCI subtypes were specifically examined in this study. It is possible that the discriminatory ability of the KDSQ may differ across MCI and dementia subtypes.

In conclusion, the KDSQ is an easily self-administered informant questionnaire that is useful for simultaneously distinguishing not only dichotomous but also multiple stages of cognitive dysfunction that are frequently encountered in the clinical setting: normal cognition, MCI, and dementia. Given its ease of use in classifying cognitive dysfunction, the KDSQ may prove useful in clinical practice.