Assessing reliability of short and tick box forms of the ANU-ADRI: Convenient alternatives of a self-report Alzheimer's disease risk assessment.

INTRODUCTION: To assess the reliability of short versions of the Australian National University Alzheimer's Disease Risk Index (ANU-ADRI).
METHODS: A short form of the ANU-ADRI (ANU-ADRI-SF) was developed by assessing risk and protective factors with single questions where possible and with short forms of sub-questionnaires where available. The tick box form of the ANU-ADRI (ANU-ADRI-TB) was developed with unique questions for each risk and protective factor for Alzheimer's disease. The short versions were evaluated in an independent community sample of 504 participants with a mean age of 45.01 (SD = 14.85, range = 18-81).
RESULTS: The short versions demonstrated high reliabilities when compared with the ANU-ADRI. However, the proportion of misclassification was high for some risk factors and particularly for the ANU-ADRI-TB. DISCUSSION: The ANU-ADRI-SF may be considered if less reliable questions from the ANU-ADRI-SF can be replaced with more reliable questions from the ANU-ADRI for risk/protective factors with high misclassification.

Introduction

At present, there is no cure or effective treatment for dementia [1], [2], [3] and existing pharmacological treatments do not modify the course of the disease [4]. Prevention is therefore one of the key objectives of current dementia research [5], and increased attention has been paid to identifying risk and protective factors for dementia. It is also essential that people understand and address their risk profile for dementia as early as possible before the development of the pathologic processes which lead to unrecoverable neurodegeneration. Easily accessible methods of risk assessment are an important tool for facilitating population-level dementia risk awareness.

A questionnaire-based risk assessment tool, the Australian National University Alzheimer's Disease Risk Index (ANU-ADRI), was developed to assess the presence of 11 risk and 4 protective factors for Alzheimer's disease (AD) [6]. These risk and protective factors have reliable scientific evidence and can be measured by self-report. The ANU-ADRI was validated on three independent cohorts [7] against the Cardiovascular Risk Factors, Aging and Dementia index and has been proposed as the key risk assessment tool in large-scale dementia prevention trials. ANU-ADRI has already been used in the first online dementia risk reduction trial in Australia [8], [9] and is currently available to the general public through the ANU-ADRI website (http://anuadri.anu.edu.au/). The website has attracted over 11,700 visitors since its launch early 2014.

The original ANU-ADRI takes approximately 15–20 minutes to complete, although longer time of up to 1 hour has been reported for older adults who were not familiar with the computer or Internet. The length of the ANU-ADRI did not affect the participation of a dementia prevention trial [9] or completion of publicly available ANU-ADRI through the website. However, a shorter version of the ANU-ADRI may have a role on public websites and online applications, enabling the ANU-ADRI to reach wider audiences in more diverse settings and in turn, save time in assessing an individual's risk level. Given that the average general practice (GP) consultation time is 14.6 minutes (95% CI, 14.1–15.0) [10], the length of the time taken to complete the current ANU-ADRI can be considered long. Hence, by providing shorter versions of the ANU-ADRI, health professionals such as GPs for example may be able to assess a client's risk level and provide them with relevant advice in a timely manner. The ANU-ADRI can also be used to screen individuals to identify those at high risk of developing AD who would benefit most from intervention programs. We therefore developed and evaluated two brief alternatives (a short form and a tick box form) to the original longer version of the ANU-ADRI while aiming to preserve its content coverage.

Methods

Participants and procedure

One thousand and seventy three people from the community were approached by Qualtrics, a survey company. Five hundred and four of them completed the original ANU-ADRI as well as one of the short versions to be validated. Sample size calculations were estimated using G*Power (version 3.1.3). To detect a medium difference (0.5) between two independent sample means, with a 5% risk of type I error (α), 95% power, and two equal groups (1:1), 105 persons in each group was required. Our sample size was more than double the required size, and half of the participants (group 1; n = 251) completed the original and the short form of the ANU-ADRI. The other half (group 2; n = 253) completed the original and the tick box form of the ANU-ADRI. Participants were randomly assigned to one of two groups by the Qualtrics survey program. Inclusion criteria were being ≥18 years, being proficient in English, having internet access, and having no psychiatric or neurological diagnoses.

Measures and procedures

Short form

A short form of the ANU-ADRI (ANU-ADRI-SF; see Supplementary A) which included the validated short forms of questionnaires used in the original ANU-ADRI (e.g. International Physical Activity Questionnaire short form [11] and 10 item Community Epidemiological Study-Depression scale (CESD-10) [12]) while keeping single-item questions (e.g., gender, age, and smoking status) was developed. The risk and protective factors covered by more than one item were also simplified into single-item questions by adding questions together. For example, two questions on diabetes “Have you ever been told by a doctor or other health professional that you have diabetes?” and “Have you ever been told by a doctor or other health professional that you have high sugar levels in your blood or urine?” became a single question “Have you ever been told by a doctor or other health professional that you have diabetes or have high sugar levels in your blood or urine?”. These single items intended to reduce the number of questions while keeping the contents. These include education, diabetes, cholesterol, traumatic brain injury, cognitive activity, and fish intake. The short form was estimated to take approximately 5 minutes to complete.

Tick box form

The tick box form of the ANU-ADRI (ANU-ADRI-TB; see Supplementary B) was built with single-item questions for each AD risk and protective factors and was estimated to take approximately 2 minutes to complete. The ANU-ADRI-TB was developed with an assumption that a single-item scale may be as sensitive and reliable as multi-item scales. This assumption was supported in previous research such as in depression [13] and physical activity [14]. It was found that a single-item interview for depressed mood provides a reliable and accurate screen that can be used in clinical settings that permit direct patient interview.

One of the short versions followed by original long version of the ANU-ADRI was administered on the same day.

Analysis

To analyze the reliability of the short versions, intra class correlation coefficients (ICC) were used between the original scores and short versions' scores. The relative measure of risk points attributed to each risk and protective factor assessed in the ANU-ADRI-SF was the same as that used in the original ANU-ADRI which ranged between 0–6 for risk factors and −7 to 0 for protective factors. ANU-ADRI-TB, however, only had binary variables (having risk/protective factors vs not) and was compared against binary variables of the original ANU-ADRI. Although the original ANU-ADRI did not evaluate misclassifications, as the tick box form can only be measured as binary outcomes (having risk/protective factors vs not), we felt it was worth examining misclassification in addition to reliability tests. The number of risk and protective factors misclassified in the short compared to the long form were calculated using binary variables (having risk/protective factors vs not). Percentages of misclassification of each risk and protective factor were also calculated to examine false-positive (being identified as having a risk factor when this is not the case) and false-negative misclassifications (not being identified as having a risk factor when this is not the case). Reliability and misclassification were not investigated for age, education, and BMI as exactly the same questions were used to assess these variables in all three versions. The data were analyzed using SPSS 22.

Results

Table 1 shows the participants' demographic characteristics and dementia risk exposure. Two hundred and forty nine (49.4%) of 504 participants were males. Participants were aged between 18 and 81 (M = 45.01, standard deviation [SD] = 14.85) years and were evenly spread across age groups of ≤30, 30–39, 40–49, 50–59, and ≥60 years. The mean age of the original validation sample was higher than the present study. However, ANU-ADRI is currently available to the general public as we are not restricting user's age. The statistics from the ANU-ADRI website demonstrated that the mean age of website users was 48.71 years (SD = 18.90). Therefore, the current sample was a better representation of the users of the ANU-ADRI.

Table 1

Demographic characteristics of participants in the short form and tick box form group

Characteristics	Short form	Tick box form	χ2	P value
Age (M, SD)	44.96 (15.03)	45.06 (14.69)	t = −0.072	.943
Gender
Females	51.0%	50.2%	0.032	.858
Education (M, SD)	14.54 (3.51)	14.94 (4.02)	t = −1.216	.225
BMI
Overweight	29.5%	29.6%	0.069	.966
Obese	28.3%	27.3%
High cholesterol	23.1%	26.1%	0.603	.437
Diabetes	13.9%	7.9%	4.727	.030
TBI	10.8%	9.9%	0.104	.747
Depression	37.1%	35.2%	0.192	.661
Physical activity
Medium	30.7 %	33.6%	3.037	.219
High	47.0%	50.2%
Cognitive activity
Middle	3.2%	3.2%	0.204	.903
Highest	31.5%	29.6%
Social activity
Low	40.6%	48.6%	5.120	.163
Fish consumption
0.25–2 p p/wk	44.2%	49.4%	4.166	.244
2–4 p p/wk	12.0%	7.5%
>4 p p/wk	6.4%	8.3%
Alcohol consumption
Light to moderate	71.5%	73.2%	0.181	.913
Smoking
Current smoker	24.3%	24.9%	2.204	.332
Past smoker	26.3%	31.6%
Pesticide	10.5%	9.2%	0.246	.620

Note. Figures are based on the original ANU-ADRI. Bold text indicates significance.

The samples from group 1 (short form) and group 2 (tick box form) were not significantly different across most characteristics. However, those who completed ANU-ADRI-SF were significantly more likely to have diabetes than those who completed ANU-ADRI-TB.

Reliability

ICC were computed to study the reliability of the short versions of the ANU-ADRI (see Table 2). ICCs for ANU-ADRI-SF suggested moderate to strong agreement with coefficients ranging from 0.772 to 0.992. All were statistically significant (P < .001) except for cognitive activity where the coefficients was small and not significant (ICC = .031, P = .285). The ANU-ADRI-TB also showed small to strong agreement for most risk factors with correlations ranging from 0.444 to 1 except for cognitive activity demonstrating extremely small and insignificant (ICC = .024, P = .287) coefficient value.

Table 2

Inter-class correlation (ICC) coefficients between short forms and long form

Risk/Protective factors	Short form	Tick box form
	ICC (95% confidence interval)	ICC (95% confidence interval)
Cholesterol	0.926 (0.905–0.942)	0.979 (0.973–0.983)
Diabetes	0.909 (0.882–0.930)	1
TBI	0.776 (0.568–0.878)	0.702 (0.509–0.819)
Depression	0.890 (0.858–0.914)	0.789 (0.725–0.838)
Physical activity	0.796 (0.737–0.841)	0.444 (0.197–0.605)
Cognitive activity	0.031 (−0.084 to 0.147)	0.024 (−0.065 to 0.118)
Social activity	0.937 (0.919–0.951)	0.822 (0.684–0.889)
Fish consumption	0.772 (0.631–0.849)	0.751 (0.667–0.812)
Alcohol consumption	0.990 (0.987–0.992)	0.640 (0.536–0.721)
Smoking	0.992 (0.990–0.994)	0.996 (0.995–0.997)
Pesticide	0.977 (0.971–0.982)	0.826 (0.774–0.865)

Bold text indicates significance.

Internal consistency was also examined for CESD-10, measuring depression, and it showed a high level of internal consistency (Cronbach's alpha of 0.879), which was comparable with the internal consistency of CESD-20 (Cronbach's alpha of 0.934).

Misclassification

Misclassification of someone having or not having risk and/or protective factors was also investigated using binary variables. Fig. 1 demonstrated that those who completed the ANU-ADRI-TB tended to have higher rates of misclassifications than those who completed the ANU-ADRI-SF.

Fig. 1

Number of risk and protective factors misclassification.

Proportions of misclassification on risk and protective factors were examined (Table 3). The false-positive misclassification represented cases where a risk or protective factor for AD was identified as present when it was not. The false-negative misclassification represented cases where a risk or protective factor for AD was identified as absent when it was not. The factor with the highest false-positive misclassification rate was cognitive activity (66.1%), whereas fish consumption (13.9%) and physical activity (13.1%) had the highest false-negative misclassification rates for the ANU-ADRI-SF. For the ANU-ADRI-TB, cognitive activity (68.4%) followed by fish consumption (11.9%) and alcohol consumption (11.5%) had the highest false-positive misclassification rates, whereas social activity level (34.4%) and physical activity level (28.1%) had the highest false-negative misclassification rates. Overall, more risk and protective factors were misclassified with ANU-ADRI-TB than ANU-ADRI-SF. Both short versions overclassified protective cognitive activity and underclassified social and physical activity.

Table 3

Percentage of misclassification

Risk/Protective factors	Short form		Tick box form		Base rate‡
	False positive∗	False negative†	False positive∗	False negative†
Cholesterol	2.8%	2%	1.2%	0.4%	23.4%
Diabetes	3.6%	0%	0%	0%	9.1%
TBI	4.0%	0%	4.3%	0.8%	6.3%
Depression	3.2%	6.0%	3.2%	11.9%	36.1%
Physical activity	4.4%	13.1%	2.0%	28.1%	81.1%
Cognitive activity	66.1%	0%	68.4%	0.4%	29.6%
Social activity	7.6%	8.8%	2.0%	34.4%	44.6%
Fish consumption	0.4%	13.9%	11.9%	1.6%	17.1%
Alcohol consumption	0.4%	0.8%	11.5%	2.4%	71.2%
Smoking	1.2%	0.4%	0.4%	0.8%	53.6%
Pesticide	0%	0.8%	0%	4.0%	9.7%

Misclassification as having risk/protective factors when they do not.

Misclassification as not having risk/protective factors when they do.

Based on the original ANU-ADRI.

Discussion

The present study developed and evaluated two short versions of the ANU-ADRI as possible alternatives to the original ANU-ADRI to be used in timely restricted settings. The short form of the ANU-ADRI demonstrated moderate to good reliability as the ANU-ADRI scores between short and long versions were similar. This suggests that when there are time constraints, the ANU-ADRI-SF may replace the original ANU-ADRI.

The ANU-ADRI-TB on the other hand had lower reliability and higher occurrence of misclassification of risk and protective factors than the ANU-ADRI-SF. Risk and protective factors that were simplified into single items or used shorter versions of existing questionnaires were likely to lead to misclassification. This suggests that factors that require comprehensive measures cannot be replaced by a single question. This appears to be especially true for factors that require a wide range of actions and/or activities (e.g., cognitive engagement) as well as frequencies and/or quantity (e.g., frequency and duration of weekly physical activity). Another possible explanation for low reliability could be the ceiling effect and that the original version had a wider range of options. For example, most adults may be engaging in at least one of the activities listed for cognitive activities every day or nearly every day. It is not surprising to find high misclassifications. Caution should therefore be exercised when using the short versions of the ANU-ADRI and other dementia risk assessment scales that do not have adequate measures of risk factors.

Translation of epidemiological research on risk factors for dementia requires the development of widely accessible assessment tools. The unique contribution of this study is to develop and evaluate a brief, practical AD risk assessment tool that can be used by clinicians or self-administered. The strength of the study was that we tested short versions of assessment tools against an established, valid tool that had been applied in practical settings. In addition, the scores were drawn from scientific evidence. However, there are some limitations to the study. The validation of the scales has not been examined as this would require following participants from when they did not have AD to when they develop AD in large populations over long periods of time to determine how well they perform in predicting AD decades later. The present study demonstrates the reliability of short versions of the ANU-ADRI when compared to the longer version. Lack of similar short versions of assessment tools also made it impossible for us to evaluate validity of the scales. Future research should investigate the validity of the short versions with large population-based samples. We also did not assess participants' cognitive functioning at the time of the assessment as this was not the aim of the present study. However, this should be considered in a future study to ensure that participants who are involved do not have any cognitive dysfunction at the time of assessment.

Overall, the present study showed that the shorter versions of the ANU-ADRI, especially the ANU-ADRI-SF have limitations but are of practical use. Short versions were least accurate in assessing lifestyle activities because these involve more complex assessment algorithms (physical, cognitive, and social activity). However, the short form is acceptable for assessing medical and demographic risk factors. Therefore, the use of the short versions should only be recommended as a second choice, when individuals have limited time and need a quick indication of an AD risk level or where lifestyle activity is not the focus of the assessment. The original ANU-ADRI cannot be fully replaced by the short versions and when using the short form, less reliable questions from the ANU-ADRI-SF should be replaced with more reliable questions from the original ANU-ADRI for those risk and protective factors with low reliability and high misclassification. We recommend using the tick box version of the ANU-ADRI only where there is no possibility of using the short form. One future possible use of the tick box form may be administration using a stepped approach, with participants being asked to complete longer questions according to the tick box version results.

Systematic review: We conducted a systematic review of the literature to identify a brief, evidence based, validated, risk assessment tool for Alzheimer's disease. No other tool that assesses level of risk for Alzheimer's or dementia has been developed that is convenient and based on self-report.

Interpretation: Our study results show that short form of the ANU-ADRI may be considered where the longer original version is not practical or when there are time constraints. However, the original ANU-ADRI remains the preferred choice. The tick box form of the ANU-ADRI is not recommended for use unless there is no other option. ANU-ADRI-TB is too brief and insufficient for measuring the complexity of cognitive, physical, and social activities.

Future directions: Further research is required to examine the usability of the ANU-ADRI-SF in various settings and feedback on effectiveness of the scale should be collected for future improvement.