Validation of a New Screening Tool for Dementia: The Simple Observation Checklist for Activities of Daily Living.

BACKGROUND AND
PURPOSE: Screening tests for dementia such as the Mini Mental State Examination (MMSE) and the Montreal Cognitive Assessment are widely used, but there are drawbacks to their efficient use. There remains a need for a brief and easy method of assessing the activities of daily living (ADL) that can be administered to elderly individuals by healthcare workers. We have therefore developed a new scale named the Simple Observation Checklist for Activities of Daily Living (SOC-ADL).
METHODS: We developed the SOC-ADL scale as a team of experts engaged in caring for individuals with dementia. This scale comprises eight items and was designed based on the Korean instrumental activities of daily living (K-IADL) scale and the Barthel activities of daily living scale (Barthel Index). The new scale was validated by enrolling 176 patients with cognitive dysfunction across 6 centers. Confirmatory factor analysis (CFA) and exploratory factor analysis (EFA) were performed. We assessed its concurrent validity by performing comparisons with the Korean-MMSE, Clinical Dementia Rating, Clinical Dementia Rating-Sum of Boxes, K-IADL, and Barthel Index, and its criterion validity by performing comparisons between mild cognitive impairment (MCI) and dementia. We also used Cronbach's alpha to assess the interitem reliability. The appropriate cutoff values were determined by analyzing receiver operating characteristic curves, including the areas underneath them.
RESULTS: EFA extracted one factor and CFA revealed that all of the model fits exceeded the minimum acceptable criteria. The SOC-ADL scores were strongly correlated with those of the other tools for dementia and could be used to differentiate MCI from dementia. Cronbach's alpha values indicated that the results were reliable. The optimal cutoff value of the SOC-ADL for discriminating dementia from MCI was 3 points, which provided a sensitivity and specificity of 74.5% and 75.7%, respectively.
CONCLUSIONS: Our results demonstrate that the SOC-ADL is a valid and reliable tool for differentiating dementia from MCI based on an assessment of ADL. This new tool can be used for screening ADL in elderly subjects who have difficulty communicating, and to increase the efficiency of dementia screening at the population level.

INTRODUCTION

The prevalence of various types of dementia including Alzheimer's disease (AD), vascular dementia (VD), and dementia with Lewy bodies (DLB) is increasing dramatically with the aging of populations worldwide. This situation has led to dementia screening programs being implemented by national policies as part of the public healthcare system in Korea since 2012, and public dementia care centers have appeared nationwide since 2018.1 The effect of massive screening for dementia is controversial due to the possibilities of overdiagnosis and a poor cost-benefit ratio;2 however, early detection is still crucial for managing dementia.3 About 80 screening tools have been developed and validated for assessing cognitive impairment using either a computer or pencil and paper,4 with the Mini Mental State Examination (MMSE) and the Montreal Cognitive Assessment being commonly used.

Korea also has a unique public monitoring system as part of the public healthcare system in which nurses or healthcare workers visit elderly individuals living alone. However, dementia screening is not regularly performed during these visits due to the burden of already administering other tests. Increasing the dementia screening rate requires consideration of the limitations of the existing public healthcare infrastructure.

For diagnosing dementia, it is necessary to evaluate the ability to perform the activities of daily living (ADL) as well as detect cognitive impairment.5 The instrumental ADL are impaired first during the early stage of dementia, which is followed by impairment of basic ADL as the disease progresses. Therefore, the evaluation of instrumental ADL is important when screening for dementia. Several scales for ADL are used in Korea, with the Korean instrumental activities of daily living (K-IADL) scale being widely applied.6 K-IADL is a well-developed and validated tool for easily assessing ADL. However, it is informant-based scale using self-reporting by the patient or by the caregiver, which means that the findings might not be entirely free of bias, including since some elderly caregivers have difficulty cooperating.7

The demand for novel, easily administered screening tools for dementia has increased in Korea. Such tools must take into account several characteristics of Korean society. The visiting programs associated with regular health screening by public health centers in Korea make it easy to check living environments. Furthermore, the dementia policies of the Korean government include detailed free neuropsychological testing for the public at dementia care centers. Therefore, there is a need to develop a novel test that focuses on ADL rather than cognition. The test should be brief and sufficiently easy so that it can be implemented by a nonexpert such as a visiting social worker or nurse.

Based on the aforementioned assessment, we developed a new screening tool named the Simple Observation Checklist for Activities of Daily Living (SOC-ADL) that allows nurses and healthcare workers to assess the ability to perform instrumental ADL. The purpose of the present study was to determine the validity and reliability of the SOC-ADL scale for clinical applications.

METHODS

Development of the SOC-ADL

The Korean Dementia Association has been funding a project named CARD (Care for ADL in dementia and Relieving symptoms in Dementia) designed to help maintain ADL in dementia patients.89 The SOC-ADL was developed as a part of the CARD project. Our task force includes medical practitioners, researchers, paramedics, nurses and psychologists who have worked in this field for more than 10 years. The task force chose to include eight items based on the K-IADL10 and the Barthel activities of daily living scale (Barthel Index),11 and revised them for use with elderly individuals who live alone or have difficulty cooperating. Each item is scored as either 0 or 1 point, and the total possible score is 8 points, with a higher score indicating better performance by the patient. The eight SOC-ADL items are as follows: 1) grooming, 2) house cleaning, 3) refrigerator use, 4) telephone use, 5) responsibility for own medications, 6) compliance with appointment dates, 7) travel, and 8) handling finances and shopping.

Study design

This multicenter, cross-sectional, observational study was designed to validate the SOC-ADL. This study enrolled 176 patients from 6 tertiary medical centers with dementia clinics. We categorized the patients into 2 groups: 102 patients had dementia and 74 patients had mild cognitive impairment (MCI).

The following inclusion criteria were applied: 1) older than 60 years, 2) diagnosed with MCI12 or degenerative early-stage dementia including AD,13 VD,14 DLB,15 and frontotemporal dementia (FTD)16 according to the corresponding clinical diagnostic criteria, 3) score of <3 points on the Clinical Dementia Rating-Sum of Boxes (CDR-SB),17 and 4) living with a caregiver. The exclusion criteria were as follows: 1) illiterate, 2) secondary dementia including normal-pressure hydrocephalus, tumor, or infection, or 3) acute comorbidity that could worsen cognition. This study was approved by the institutional review board at each participating center (IRB number: HYUH 2017-09-006). Patients and their caregivers provided written informed consents.

Outcome measurements

Demographic factors including age, sex, education level, alcohol history, smoking history, caregiver information, and medical history were obtained. All participants completed the Korean-MMSE (K-MMSE),18 Clinical Dementia Rating (CDR),19 and CDR-SB, and underwent an imaging investigation (brain CT and MRI) for a diagnosis of dementia. After being diagnosed with MCI or dementia, the participants completed the SOC-ADL as well as the K-IADL and Barthel Index. Concurrent validity was analyzed using the K-IADL score as calculated using the formula (total score for 11 items, excluding not-applicable items), rather than the total score.

Statistical analysis

We used AMOS (version 20.0, SPSS, Chicago, IL, USA) to perform the confirmatory factor analysis (CFA), SPSS Statistics (version 25, IBM Corp., Armonk, NY, USA) for the exploratory factor analysis (EFA) and other statistical analyses, and R software (version 3.6.2, The R foundation) for comparing receiver operating characteristic (ROC) curves. The data are presented as frequency (percentage) values for categorical variables and mean±standard-deviation values for continuous variables. A p value of <0.05 was considered statistically significant.

We used both CFA and EFA to assess construct validity. Whether the factor structure could be replicated in the new data set from 176 patients was investigated using EFA with maximum-likelihood factoring and CFA. The sampling adequacy was evaluated using a Kaiser-Mayer-Olkin (KMO) test and Bartlett test, with a KMO value of >0.7 considered acceptable. To assist in interpreting the factors, we used an orthogonal varimax rotation, which assumes that the factors are uncorrelated. The subjective screening test uses a scatter plot of eigenvalues—in which their ranks are plotted versus their magnitudes—to extract as many factors as there are eigenvalues that fall before the last large drop (i.e., an elbow shape) in the plot. Once the factors were chosen, an item was retained as a factor if its factor loading was ≥0.4.

For CFA, we evaluated the chi-square (χ2) value, Tucker-Lewis index (TLI), standard root-mean-square residual (SRMR), comparative fit index (CFI), and root-mean-square error of the approximation (RMSEA) as model fit indices. The model adequacy was tested in the CFA using commonly accepted indices designed to evaluate model fits. The following values of the fit indices were considered to indicate acceptable model adjustment: normed χ2 [chi-square minimum/degree of freedom (CMIN/DF)] ≤3.0, RMSEA <0.08, CI 95%), TLI >0.90, SRMR <0.05, and CFI >0.90.

To assess concurrent validity, we quantified the correlations of the SOC-ADL with the K-MMSE, CDR, CDR-SB, K-IADL, and Barthel Index by calculating Spearman's correlation coefficients. For assessing criterion validity, we compared each item of the SOC-ADL between MCI and dementia patients. Ranked analysis of covariance [ranked analysis of covariance (ANCOVA)] was used to assess criterion validity with adjustment of different demographic factors. In order to test the internal consistency of the items, Cronbach's alpha values were calculated for the SOC-ADL, with values >0.7 considered acceptable.

Finally, we derived optimal cutoff scores that satisfied both sensitivity and specificity criteria for dementia using ROC analysis. To assess the diagnostic accuracy of each test for dementia, we compared the areas under the ROC curves (AUCs) of the SOC-ADL, CDR, CDR-SB, and K-IADL using Delong's method.20

RESULTS

Construct validity

The demographic data are compared between MCI and dementia in Table 1. In the dementia group, 91 patients were diagnosed with AD, 9 with VD, and 1 with FTD. Sex, age, and smoking history differed between MCI and dementia.

Table 1

Demographic factors in all subjects

	Number (%)
Sex
Male	87 (49.4)
Female	89 (50.6)
Age (yr)
50s	4 (2.3)
60s	21 (11.9)
70s	90 (51.1)
80s	59 (33.3)
90s	2 (1.1)
Education level (yr)
0	2 (1.1)
0.5–5	39 (22.0)
6-8	62 (35.2)
9–11	25 (14.2)
12–15	34 (19.3)
≥16	14 (8.0)
Alcohol history
Heavy drinking	6 (3.4)
Social drinking	41 (23.3)
No drinking	129 (73.3)
Smoking history
Nonsmoker	131 (74.4)
Past smoker	36 (20.5)
Current smoker	9 (5.1)
Caregiver
Spouse	67 (38.1)
Son or daughter	77 (43.8)
Sibling	1 (0.6)
Other relative	-
Not a relative	17 (9.7)
Diagnosis
Mild cognitive impairment	74 (42.0)
Alzheimer's disease	91 (51.7)
Vascular dementia	10 (5.6)
Frontotemporal dementia	-
Dementia with Lewy bodies	1 (0.6)
Total	176 (100.0)

In the EFA, a one-factor solution was considered and analyzed using the common-factor model. The eigenvalues identified in the principal-components analysis were 2.908, 0.969, 0.849, 0.793, 0.643, 0.570, and 0.441, indicating a sharp drop after the first factor (Fig. 1). Table 2 lists the factor loadings for the one-factor model. The first factors that explained 36.4% of the total variance comprised grooming, house cleaning, refrigerator use, telephone use, responsibility for own medications, compliance with appointment dates, travel, and handling finances and shopping, whose r values for the factor loadings were 0.524, 0.720, 0.660, 0.408, 0.647, 0.468, 0.619, and 0.702, respectively.

Fig. 1

Scree plots in exploratory factor analysis. The number of factors is 1, which have eigenvalues greater than 1.0. A sharp drop in eigenvalues after the first factor was found.

Table 2

Results of the confirmatory factor analysis

Parameter	χ2	DF	Normed χ2 (CMIN/DF)	TLI	SRMR	CFI	RMSEA (LO, UP)
Measurement index	23.565	20	1.178	0.977	0.0406	0.983	0.032 (0.000, 0.075)
Criterion			≤3.0	≥0.90	≤1.0	≥0.90	≤0.08

LO and UP are the limits of the 95% confidence interval of RMSEA.

CFI: comparative fit index, CMIN: chi-square minimum, DF: degree of freedom, LO: lower limit, RMSEA: root-mean-square error of the approximation, SRMR: root-mean-square residual, TLI: Tucker-Lewis index, UP: upper limit.

CFA was used to test the factors found in the EFA. Table 2 presents the results for the goodness-of-fit indices of the measurement model. All of the fit indices for this one-factor model were satisfactory (χ2=23.565, DF=20, CMIN/DF=1.178, TLI=0.977, SRMR=0.0406, CFI=0.983, and RMSEA= 0.032).

Concurrent validity, criterion validity, and interitem reliability

We used Spearman's correlation to assess the concurrent validity of the SOC-ADL in comparison with existing dementia scales. The SOC-ADL was significantly correlated with the K-MMSE (r=−0.414, p<0.001), CDR (r=0.617, p<0.001), CDR-SB (r=0.723, p<0.001), K-IADL total score (r=0.753, p<0.001), K-IADL scores for each item (r=0.706, p<0.001), and the Barthel Index (r=−0.346, p<0.001), indicating that the SOC-ADL has a high concurrent validity (Table 3).

Table 3

Concurrent validity of SOC-ADL by comparisons with other screening tools for dementia

	K-MMSE score	CDR score	CDR-SB score	K-IADL score	Barthel Index
SOC-ADL total score	-0.414 (<0.001)	0.617 (<0.001)	0.723 (<0.001)	0.706 (<0.001)	-0.346 (<0.001)
Grooming	-0.246 (<0.001)	0.250 (<0.001)	0.307 (<0.001)	0.259 (<0.001)	-0.179 (0.017)
House cleaning	-0.220 (0.003)	0.390 (<0.001)	0.497 (<0.001)	0.457 (<0.001)	–0.228 (0.002)
Refrigerator use	-0.357 (<0.001)	0.435 (<0.001)	0.479 (<0.001)	0.474 (<0.001)	-0.270 (<0.001)
Telephone use	-0.163 (0.030)	0.136 (0.072)	0.259 (<0.001)	0.279 (<0.001)	-0.197 (0.009)
Responsibility for own medications	-0.251 (<0.001)	0.302 (<0.001)	0.410 (<0.001)	0.514 (<0.001)	-0.226 (0.003)
Compliance with appointment dates	-0.152 (0.043)	0.300 (<0.001)	0.399 (<0.001)	0.228 (0.002)	-0.166 (0.028)
Travel	-0.194 (0.010)	0.396 (<0.001)	0.487 (<0.001)	0.391 (<0.001)	-0.354 (<0.001)
Handling finances and shopping	-0.387 (<0.001)	0.547 (<0.001)	0.517 (<0.001)	0.620 (<0.001)	-0.174 (0.021)

CDR: Clinical Dementia Rating, CDR-SB: Clinical Dementia Rating-Sum of Boxes, K-IADL: Korean instrumental activities of daily living, K-MMSE: Korean-Mini Mental State Examination, SOC-ADL: Simple Observation Checklist for Activities of Daily Living.

For criterion validity, we performed the known-group method in which items are compare between two known groups. We used ranked ANCOVA to adjust different factors (sex, age, and smoking history). The SOC-ADL showed a significant ability to differentiate the MCI and dementia groups. The following subitems differed significantly between the two groups: house cleaning (p=0.001), refrigerator use (p<0.001), responsibility for own medications (p<0.001), compliance with appointment dates (p<0.001), travel (p<0.001), and handling finances and shopping (p<0.001). The scores on all scales including the K-MMSE, CDR, CDR-SB, K-IADL, and Barthel Index differed significantly between MCI and dementia patients (Table 4).

Table 4

Criterion validity of SOC-ADL by comparisons between MCI and dementia

	Total (n=176)	Diagnosis		p
		Dementia (n=102)	MCI (n=74)
K-MMSE score	21.17±3.66	19.84±3.16	23.00±3.53	<0.001
CDR score	0.75±0.26	0.88±0.21	0.57±0.20	<0.001
CDR-SB score	3.69±2.07	4.65±1.75	2.38±1.75	<0.001
K-IADL score	0.91±2.12	0.89±0.51	0.93±3.23	<0.001
Barthel Index	19.72±0.81	19.61±0.94	19.86±0.56	0.030
SOC-ADL	2.85±2.09	3.70±1.96	1.68±1.64	<0.001
Grooming	0.06±0.24	0.10±0.30	0.01±0.12	0.023
House cleaning	0.32±0.47	0.42±0.50	0.19±0.39	0.001
Refrigerator use	0.37±0.48	0.54±0.50	0.14±0.34	<0.001
Telephone use	0.14±0.35	0.16±0.37	0.12±0.33	0.510
Responsibility for own medications	0.50±0.50	0.62±0.49	0.34±0.48	<0.001
Compliance with appointment dates	0.76±0.43	0.90±0.30	0.57±0.50	<0.001
Travel	0.26±0.44	0.36±0.48	0.12±0.33	<0.001
Handling finances and shopping	0.43±0.50	0.60±0.49	0.19±0.39	<0.001

Data are mean±standard-deviation values. p values are from the Mann-Whitney U test. The Shapiro-Wilk test was used to check for normality.

CDR: Clinical Dementia Rating, CDR-SB: Clinical Dementia Rating-Sum of Boxes, K-IADL: Korean instrumental activities of daily living, K-MMSE: Korean-Mini Mental State Examination, MCI: mild cognitive impairment, SOC-ADL: Simple Observation Checklist for Activities of Daily Living.

Cronbach's alpha for the interitem consistency of the SOC-ADL was 0.743, which indicates that the test has acceptable reliability.

ROC analysis

We used ROC curve analysis to determine the optimal cut-off values for discriminating MCI and dementia. The sensitivity, specificity, positive predictive value, and negative predictive value were calculated for particular cutoff values. The dichotomization based on a score of 2.5 points on the SOC-ADL produced statistically significant results as a diagnostic tool for differentiating dementia from MCI (Fig. 2), with a sensitivity of 74.5% and a specificity of 75.7% (Table 5).

Fig. 2

The ROC curve and AUC of the SOC-ADL and comparison with other scales. AUC of SOC-ADL (A). Comparison of AUC of the SOC-ADL, CDR, CDR-SB and K-IADL (B). ADL: activities of daily living, AUC: area under the curve, CDR: Clinical Dementia Rating, CDR-SB: Clinical Dementia Rating-Sum of Boxes, K-IADL: Korean instrumental activities of daily living, ROC: receiver operating characteristic curve, SOC-ADL: Simple Observation Checklist for Activities of Daily Living.

Table 5

Results of AUC analysis of the Simple Observation Checklist for Activities of Daily Living in predicting dementia

Cut-off	Diagnosis		AUC (p)	Sensitivity (%)	Specificity (%)	PPV (%)	NPV (%)
	Diagnosis	MCI
≥2.50	56.0	26.0	0.791 (<0.001)	74.5	75.7	80.9	68.3
<2.50	18.0	76.0

Sensitivity, 76/102; specificity, 56/74; PPV, 76/94; NPV, 56/82; false positives, 24.3% (18/74); false negatives, 25.5% (26/102).

AUC: area under the curve, MCI: mild cognitive impairment, NPV: negative predictive value, PPV: positive predictive value.

The AUCs of the SOC-ADL (AUCSOC-ADL), CDR (AUCCDR), CDR-SB (AUCCDR-SB), and K-IADL (AUCK-IADL) all exceeded 0.79, indicating that all of the tests are useful in detecting dementia. AUCSOC-ADL did not differ significantly from AUCCDR (z=−0.362, p=0.717), AUCCDR-SB (z=−1.714, p=0.087), or AUCK-IADL (z=−0.227, p=0.821) (Fig. 2).

DISCUSSION

The SOC-ADL scale was designed for use by visiting healthcare workers and nurses, and it serves an unmet need for elderly individuals who have difficulty communicating. The present results demonstrate that the SOC-ADL scale has acceptable validity for differentiating dementia from MCI, based on statistical analyses including EFA, CFA, concurrent validity, criterion validity, and interitem reliability (Cronbach's alpha). Both CFA and EFA demonstrated that the SOC-ADL was satisfactory. In the CFA, the SOC-ADL showed acceptable results for all measurement models, while the EFA showed only one factor in its item distribution.

The SOC-ADL scale was designed for use by visiting healthcare workers and nurses, and it serves an unmet need for elderly individuals who have difficulty communicating. The present results demonstrate that the SOC-ADL scale has acceptable validity for differentiating dementia from MCI, based on statistical analyses including EFA, CFA, concurrent validity, criterion validity, and interitem reliability (Cronbach's alpha). Both CFA and EFA demonstrated that the SOC-ADL was satisfactory. In the CFA, the SOC-ADL showed acceptable results for all measurement models, while the EFA showed only one factor in its item distribution. the Korean version of Lawton ADL scale,21 the Seoul IADL scale,22 and the Korean version of the Disability Assessment for Dementia scale.23 Each of these scales has strengths and weaknesses. The most striking advantage of the SOC-ADL is that nonexperts can administer it rapidly, with clear evidence that it is strongly correlated with other scales. Another strength is that the score of the SOC-ADL is easy to calculate. However, some caution is required when using the SOC-ADL, with examiners needing to consider the socioeconomic status (item 8), residing area (item 7), and gender roles (items 2 and 3) of the patient.

Direct interviews are not mandatory for measuring the eight items in the SOC-ADL, since the required information can be obtained via self-reporting from patients or caregivers. Visiting nurses can check items 1 (grooming), 2 (house cleaning), 3 (refrigerator use), and 5 (responsibility for own medications) by merely observing the home situation of the patient. Items 4 (telephone use), 6 (compliance with appoint ment dates), 7 (travel), and 8 (handling finances and shopping) can also be readily checked without difficulty. These characteristics make the SOC-ADL easy to apply to illiterate elderly individuals who have severe cognitive dysfunction, are living alone or with a caregiver, or have difficulty communicating.

The total score of the SOC-ADL was correlated most strongly with the K-IADL and most weakly with the Barthel Index. This might be because the Barthel Index is designed for the overall ADL function in all geriatric diseases, whereas the K-IADL focuses on early-stage dementia. In fact, the Barthel Index is usually normal in AD patients but not in other types of dementia.24 The SOC-ADL score is therefore preferred over the Barthel Index for patients with early-stage dementia.

Handling finances and shopping was the SOC-ADL item that showed the strongest correlation with the other scales. Checking account books or banking records is the most-sensitive item for estimating ADL. For criterion validity, all items other than telephone use (p=0.510) differed significantly between MCI and dementia. The telephone-use item covered a broad range of definitions that included landline, cellular, and smart phones, and so this question might be affected by performance bias.

Our results indicated that the SOC-ADL has a broader application scope as a screening tool for dementia. Systems for screening dementia are usually affected by poor accessibility, which reduces their adoption in medical institutions and public healthcare centers because many elderly patients experience difficulties walking due to medical illnesses. This situation makes screening dementia more difficult in rural than urban areas. Visiting healthcare services can overcome the medical blind spot in dementia screening in rural areas by utilizing a test such as the SOC-ADL. Our results demonstrated the SOC-ADL can be a valid screening tool for use by visiting healthcare services, and hence it can help to close the gap in dementia screening between rural and urban areas. Furthermore, the utility of the SOC-ADL can be extended to underdeveloped countries that lack public transportation.

This study was subject to a few limitations. First, we dichotomized all types of dementia and MCI into two groups: dementia vs. MCI. We clustered all types of dementia into a single group to simplify the SOC-ADL because it was designed as a screening tool for nonexperts. Second, the proportion of dementia types in this study does not reflect the actual proportions of the various types of dementia in Korea, since most of the patients with dementia had AD (89%) and none of them had FTD. Future studies should investigate the validity and reliability of the SOC-ADL for each type of dementia. Third, the SOC-ADL still has informant bias despite the use of a simple questionnaire. The SOC-ADL comprises items that can be obtained using easy questions. However, some items such as compliance with appointment dates, travel, and handling finances and shopping are difficult to check in a single visit.

In conclusion, the SOC-ADL score comprising eight items enables nonexperts to rapidly and objectively measure the ADL status. This tool is thus a candidate for public dementia screening in Korea. Original Korean version of SOC-ADL is posted in Supplementary Materials (Supplementary Materials in the online-only Data Supplement).