Prediction of dementia risk in low-income and middle-income countries (the 10/66 Study): an independent external validation of existing models.

BACKGROUND: To date, dementia prediction models have been exclusively developed and tested in high-income countries (HICs). However, most people with dementia live in low-income and middle-income countries (LMICs), where dementia risk prediction research is almost non-existent and the ability of current models to predict dementia is unknown. This study investigated whether dementia prediction models developed in HICs are applicable to LMICs.
METHODS: Data were from the 10/66 Study. Individuals aged 65 years or older and without dementia at baseline were selected from China, Cuba, the Dominican Republic, Mexico, Peru, Puerto Rico, and Venezuela. Dementia incidence was assessed over 3-5 years, with diagnosis according to the 10/66 Study diagnostic algorithm. Discrimination and calibration were tested for five models: the Cardiovascular Risk Factors, Aging and Dementia risk score (CAIDE); the Study on Aging, Cognition and Dementia (AgeCoDe) model; the Australian National University Alzheimer's Disease Risk Index (ANU-ADRI); the Brief Dementia Screening Indicator (BDSI); and the Rotterdam Study Basic Dementia Risk Model (BDRM). Models were tested with use of Cox regression. The discriminative accuracy of each model was assessed using Harrell's concordance (c)-statistic, with a value of 0·70 or higher considered to indicate acceptable discriminative ability. Calibration (model fit) was assessed statistically using the Grønnesby and Borgan test.
FINDINGS: 11 143 individuals without baseline dementia and with available follow-up data were included in the analysis. During follow-up (mean 3·8 years [SD 1·3]), 1069 people progressed to dementia across all sites (incidence rate 24·9 cases per 1000 person-years). Performance of the models varied. Across countries, the discriminative ability of the CAIDE (0·52≤c≤0·63) and AgeCoDe (0·57≤c≤0·74) models was poor. By contrast, the ANU-ADRI (0·66≤c≤0·78), BDSI (0·62≤c≤0·78), and BDRM (0·66≤c≤0·78) models showed similar levels of discriminative ability to those of the development cohorts. All models showed good calibration, especially at low and intermediate levels of predicted risk. The models validated best in Peru and poorest in the Dominican Republic and China.
INTERPRETATION: Not all dementia prediction models developed in HICs can be simply extrapolated to LMICs. Further work defining what number and which combination of risk variables works best for predicting risk of dementia in LMICs is needed. However, models that transport well could be used immediately for dementia prevention research and targeted risk reduction in LMICs. FUNDING: National Institute for Health Research, Wellcome Trust, WHO, US Alzheimer's Association, and European Research Council.

Introduction

Dementia is a substantial global health issue. Reduction in future numbers of dementia cases through effective preventive strategies could significantly affect the personal and socioeconomic burdens of dementia. WHO has recommended that countries urgently develop national public health programmes to reduce the impact of dementia, and has also published specific guidelines for governments, policy makers, and care providers to develop and deliver a public health approach to dementia prevention. Such an approach is particularly essential in low-income and middle-income countries (LMICs), which are home to about two-thirds of the people living with dementia globally, but have lower availability of the resources needed to cope with dementia-related care compared with high-income countries (HICs). For research to move rapidly towards prevention, and eventually to limit the expected increase in dementia rates in LMICs, it is necessary to establish methods for early and accurate identification of individuals at high risk of future dementia.

More than 20 different models for predicting dementia risk have been developed in HICs.3, 4, 5 Predictive accuracy, measured using the concordance (c)-statistic (a measure of the probability that a randomly selected person with the outcome of interest has a higher risk of the outcome than a randomly selected person who did not experience the outcome of interest), has been found to range from poor (c=0·48) to high (c=0·91). Very few models have been externally validated and, where tested, results have been mixed and limited to assessment in HICs only.8, 9 Methodological differences, such as time of testing (eg, mid-life [45–64 years] vs later life [≥65 years]) or length of follow-up, as well as differing population characteristics (eg, age and distribution of risk factors) between the development and validation samples probably account for the poor external validity.8, 9

Evidence before this study

Our group has published a number of systematic reviews summarising the evidence on the development and testing of models for predicting future dementia. These reviews show that more than 20 models have been developed for predicting risk of future dementia. The discriminative ability of the models has been found to be variable (with concordance [c]-statistics ranging from 0·48 to 0·91). However, model development and testing for dementia risk prediction has only been undertaken in high-income countries (HICs). Furthermore, where tested, few models have been found to have reasonable predictive accuracy outside the setting in which they were developed (ie, external validity). To date, no external validation studies have been undertaken in low-income and middle-income countries (LMICs). Therefore, it is unknown whether models derived from cohorts in HICs can be applied in LMIC settings.

Added value of this study

Using data from the 10/66 Study, we investigated whether dementia risk prediction models developed in HICs can be applied in LMICs without compromise to their level of discriminative accuracy. Five models were tested: the Cardiovascular Risk Factors, Aging and Dementia risk score (CAIDE), the Study on Aging, Cognition and Dementia (AgeCoDe) model, the Australian National University Alzheimer's Disease Risk Index (ANU-ADRI), the Brief Dementia Screening Indicator (BDSI), and the Rotterdam Study Basic Dementia Risk Model (BDRM). The results were mixed. The CAIDE and the AgeCoDe models showed poor discriminative ability when applied in LMICs. By contrast, in all countries, the ANU-ADRI, BDSI, and BDRM showed similar performance to when they were mapped in HIC settings. The models were well calibrated, especially at low and intermediate risk levels.

Implications of all the available evidence

Current dementia risk prediction models developed in HICs, including the ANU-ADRI, BDSI, and BDRM, can be used in LMICs without compromise to their discriminative ability. Importantly, these models provide an effective and simple way of identifying individuals who would benefit from intervention to reduce their dementia risk. However, further work is needed to undertake model development and testing in LMICs to assess whether predictive accuracy and calibration can be improved. Such models are necessary to move rapidly towards dementia prevention, and eventually limit the expected increase in dementia rates in LMICs.

It is unknown whether models developed in HICs are applicable and accurate for dementia risk prediction in LMICs. Therefore, we aimed to externally validate dementia prediction models developed in HICs by applying them in LMICs, including sites in Latin America, the Caribbean, and China, using data from the 10/66 Study.10, 11

Methods

Study design and participants

Data were from the 10/66 Study, the protocols for which have been published elsewhere.10, 11 At baseline, one-phase population-based surveys (2004–06 in all sites except Puerto Rico, where baseline was 2007–10) of all individuals aged 65 years or older, living in geographically defined catchment areas in ten countries (Brazil, China, Cuba, Dominican Republic, India, Mexico, Nigeria, Peru, Puerto Rico, and Venezuela), were conducted. Sample size ranged from 1900 to 3000 across countries, and more than 80% of the target population responded in all areas surveyed. At baseline, participants completed a comprehensive interview that included information on household, sociodemographic, and health status, and a physical and neurological exam including the Geriatric Mental State (GMS) examination. Informants (which could include caregivers, co-residents, family, or other close contacts) were also interviewed, where available. Blood samples were collected in all countries except China.

Follow-up interviews were done in seven countries (China, Cuba, Dominican Republic, Mexico, Peru, Puerto Rico, and Venezuela), at approximately 3–5 years from baseline (2007–10 for all countries except Puerto Rico where follow-up was 2012–13). At follow-up, the baseline assessment was repeated. Data from these seven countries were used in this analysis.

Participants or their informant (where participants lacked the capacity to consent) gave written consent. The 10/66 Study has institute (Institute of Psychiatry, King's College London, London, UK) and local ethics committee approval.

Selection of risk models for external validation

Dementia prediction models were selected from recent systematic reviews3, 4, 5 using four criteria: sufficient information was published to allow calculation of individual risk scores, including predictor weights; identical or similar predictor variables were available in the 10/66 Study dataset to enable accurate mapping of the risk score; the variables included in the risk model were simple to attain (eg, we excluded models that incorporated neuroimaging data); and the predictive accuracy of the model, as reported in the development study, was acceptable (defined as a c-statistic ≥0·70). On the basis of these criteria, five models were selected: the Cardiovascular Risk Factors, Aging and Dementia (CAIDE) risk score; the Aging, Cognition and Dementia (AgeCoDe) model; the Australian National University Alzheimer's Disease Risk Index (ANU-ADRI);16, 17 the Brief Dementia Screening Indicator (BDSI); and the Rotterdam Study Basic Dementia Risk Model (BDRM). Table 1 includes full details of each model (see appendix 3 [pp 1–2] for information on how the risk variables incorporated in the different models were assessed in the 10/66 Study).

Table 1

Description of the five models validated in the 10/66 Study

	Country	Sample size	Age at baseline, years	Follow-up, years	Model components	Modifications made in the 10/66 Study*	Outcome	Incident dementia, %	c-statistic (95% CI)
CAIDE model (Kivipelto et al, 2006)14
Cardiovascular Risk Factors, Aging, and Dementia	Finland	1409	39–64	Mean 20·0 (SD 4·9)	Age, sex, education, systolic blood pressure, BMI, total cholesterol, and physical activity	BMI replaced by waist-to-height ratio; education level adjusted to years; physical activity level was re-grouped to be binary	Dementia (DSM-IV)	4%	0·77 (0·71–0·83)
AgeCoDe model (Jessen et al, 2011)15
The Aging, Cognition and Dementia Study	Germany	3055	≥75	Mean 3·81 (max 6·14)	Age, subjective memory impairment, verbal fluency, delayed recall, MMSE, and IADL	Subjective memory impairment scores re-grouped; MMSE replaced with CDR; number of ADL or IADL adjusted to IADL group	Alzheimer's disease (DSM-IV, NINCDS -ADRDA)	6·3%	0·84 (0·80–0·88)
ANU-ADRI (Anstey et al, 2014)16, 17
Cardiovascular Health Cognition Study	USA	2496	≥62	Median 6·0	Age group (by sex), education, BMI, diabetes, symptoms of depression, total cholesterol, traumatic brain injury, smoking, alcohol use, social engagement, physical activity, cognitive activity, fish intake, and pesticide exposure	BMI replaced by waist-to-height ratio; score mapped without pesticide exposure or cognitive activity	Dementia (study-specific criteria)	11·1%	0·73 (0·70–0·75)
Kungsholmen Project	Sweden	905	≥74	Mean 6·0 (SD 5·7)	As above	As above	Dementia (DSM-III revised)	20·0%	0·65 (0·62–0·69)
Rush Memory and Aging Project	USA	903	≥54	Mean 3·5 (SD 3·0)	As above	As above	Dementia (NINCDS-ADRDA)	17·5%	0·72 (0·68–0·76)
BDSI (Barnes et al, 2014)18
Cardiovascular Health Study	USA	2794	≥65	6	Age, education, BMI, diabetes, stroke, IADL (needs help with money or medications), and depressive symptoms	BMI replaced by waist-to-height ratio; IADL (needs help with handling money only); diabetes and stroke were self-reported	Dementia (study-specific criteria)	14·2%	0·68 (0·60–0·72)
Framingham Heart Study	USA	2411	≥65	6	As above	As above	Dementia (DSM-IV)	7·1%	0·77 (0·73–0·82)
Health and Retirement Study	USA	13 889	≥65	6	As above	As above	Dementia (cutoff point on a brief cognitive battery)	15·6%	0·76 (0·74–0·77)
Sacramento Area Latino Study on Aging	USA	1125	≥65	6	As above	As above	Dementia (study-specific criteria)	7·6%	0·78 (0·72–0·83)
BDRM (Licher et al, 2019)19
Rotterdam Study	Netherlands	2710	60–96	Median 7 (IQR 5·1–9·1)	Age, stroke, subjective memory decline, IADL (needs help with finances or medication)	Subjective memory decline measured by subjective memory impairment score; IADL (needs help with handling money only)	Dementia (DSM-III revised)	4·8%	0·78 (0·75–0·81)
Epidemiological Prevention Study of Zoetermeer	Netherlands	514	60–96	9·5 (IQR 7·6–11·4)	As above	As above	Dementia (DSM-III revised)	7·0%	0·75 (0·67–0·82)

ADL=activities of daily living. AgeCoDe=Study on Aging, Cognition and Dementia. ANU-ADRI=Australian National University Alzheimer's Disease Risk Index. BDRM=Rotterdam Study Basic Dementia Risk Model. BDSI=Brief Dementia Screening Indicator. BMI=body-mass index. CAIDE=Cardiovascular Risk Factors, Aging, and Dementia Risk Score. CDR=Clinical Dementia Rating Scale (sum of boxes). c-statistic=concordance statistic. DSM=Diagnostic and Statistical Manual of Mental Disorders. IADL=instrumental activities of daily living. MMSE=Mini-Mental State Examination. NINCDS-ADRDA=National Institute of Neurological and Communicative Disorders and Stroke and the Alzheimer's Disease and Related Disorders Association.

Full details are provided in the appendix (pp 3–5).

Outcome

The outcome was incident dementia (all causes), diagnosed according to the 10/66 diagnostic algorithm. This probabilistic algorithm incorporates cognitive test scores, including the Community Screening Instrument for Dementia (CSI-D) COGSCORE and the modified Consortium to Establish a Registry for Alzheimer's Disease ten-word list-learning task with delayed recall, as well as informant reports of cognitive and functional decline from the CSI-D RELSCORE and diagnostic output from the GMS examination.

For the analysis, time to diagnosis was defined as the date from the baseline interview to the date of the follow-up interview when dementia was diagnosed. A prespecified sensitivity analysis was completed with time of diagnosis defined as the midway point between the baseline and follow-up visits.

Statistical analysis and external validation

All analyses were done with Stata version 15.0. The study was conducted and reported in line with the Transparent Reporting of a Multivariate Prediction Model for Individual Prediction or Diagnosis statement.

Differences between countries in sociodemographic and health variables were compared using ANOVA for continuous variables and χ2 test for categorical variables.

We computed the individual probabilities for each of the five risk models using the original prediction algorithms. Full details of how each risk model was mapped in the 10/66 Study, including the scores allocated to each variable, are shown in the appendix 3 (pp 3–5). As blood samples were not collected in China, information on total cholesterol (required for mapping the CAIDE and ANU-ADRI scores) was unavailable. Therefore, the CAIDE and ANU-ADRI models were mapped in China without cholesterol data.

Cox proportional hazards regression was used to test each model. Cox regression was chosen to allow us to account for time to event and censoring (ie, death and dropout). Time on the study was taken as the time from the baseline assessment to diagnosis of dementia, end of follow-up, or death or dropout, whichever came first. We used the Schoenfeld residuals test to check the proportional hazards assumption and found no violations. Discriminative accuracy was assessed using Harrell's c-statistic. Calibration, or model fit, was assessed statistically using the Grønnesby and Borgan (GB) test and graphically using Cox-Snell residual plots. Models were tested in each country separately. In order to quantify overall (average) predictive performance and heterogeneity (I2 statistic) in predictive performance across the different countries, we also combined the c-statistic outputs with use of a random-effects meta-analysis.

Individuals with missing data were excluded.

Role of the funding source

The funders had no role in study design or conduct, data collection, data analysis, data interpretation, writing of the report, or submission of the manuscript for publication. MPrina had full access to all the data. BCMS, LR, and MPrina took the decision, in conjunction with the coauthors, to submit the manuscript for publication.

Results

At baseline, 15 016 participants were recruited, of whom 1429 with prevalent dementia and 2444 who were not seen at follow-up were excluded. Thus, the analysed sample included 11 143 participants (6973 [62·6%] women and 4164 [37·4%] men [of 11 137 with non-missing data]), with a mean age of 73·8 years (SD 6·6; range 65–106). There were significant differences in the sociodemographic characteristics of the samples across the different study sites at baseline (table 2).

Table 2

Baseline characteristics

		Total sample	Country-specific data							p value
			Cuba	Dominican Republic	Peru	Venezuela	Mexico	Puerto Rico	China
Cohort characteristics
Sample size at baseline		15 016	2944	2011	1933	1965	2003	1998	2162	..
Dementia at baseline		1429/15 016 (9·5%)	323/2944 (11·0%)	242/2011 (12·0%)	166/1933 (8·6%)	145/1965 (7·4%)	180/2003 (9·0%)	233/1998 (11·7%)	140/2162 (6·5%)	..
Missing follow-up status		2444/15 016 (16·3%)	321/2944 (10·9%)	328/2011 (16·3%)	444/1933 (23·0%)	467/1965 (23·8%)	302/2003 (15·1%)	392/1998 (19·6%)	190/2162 (8·8%)	..
Sample used for analysis		11 143/15 016 (74·2%)	2300/2944 (78·1%)	1441/2011 (71·7%)	1323/1933 (68·4%)	1353/1965 (68·9%)	1521/2003 (75·9%)	1373/1998 (68·7%)	1832/2162 (84·7%)	..
Follow-up status
Mean follow-up time, years		3·8 (1·2)	4·0 (1·3)	4·3 (1·5)	3·0 (0·8)	3·9 (1·1)	2·8 (0·6)	4·0 (1·2)	4·4 (1·2)	<0·0001
Dementia at follow-up		1069/11 143 (9·6%)	182/2300 (7·9%)	165/1441 (11·5%)	77/1323 (5·8%)	155/1353 (11·5%)	130/1521 (8·5%)	153/1373 (11·1%)	207/1832 (11·3%)	<0·0001
Deceased at follow-up		1709/11 143 (15·3%)	437/2300 (19·0%)	323/1441 (22·4%)	101/1323 (7·6%)	141/1353 (10·4%)	157/1521 (10·3%)	170/1373 (12·4%)	380/1832 (20·7%)	<0·0001
Demographic characteristics at baseline*
Sex										<0·0001
	Female	6973/11 137 (62·6%)	1480/2300 (64·4%)	948/1439 (65·9%)	791/1323 (59·8%)	849/1353 (62·7%)	963/1521 (63·3%)	923/1369 (67·4%)	1019/1832 (55·6%)	..
	Male	4164/11 137 (37·4%)	820/2300 (35.7%)	491/1439 (34·1%)	532/1323 (40·2%)	504/1353 (37·3%)	558/1521 (36·7%)	446/1369 (32·6%)	813/1832 (44·4%)	..
Age, years		73·8 (6·6)	74·4 (6·6)	74·6 (7·2)	74·2 (7·0)	71·8 (6·4)	73·6 (6·2)	75·4 (6·5)	72·8 (5·9)	..
Education										<0·0001
	Primary school not completed	4468/11 107 (40·2%)	531/2295 (23·1%)	1019/1435 (71·0%)	256/1313 (19·5%)	391/1344 (29·1%)	1047/1519 (68·9%)	273/1369 (19·9%)	951/1832 (51·9%)	..
	Completed primary school	3231/11 107 (29·1%)	771/2295 (33·6%)	266/1435 (18·5%)	493/1313 (37·5%)	683/1344 (50·8%)	286/1519 (18·8%)	260/1369 (19·0%)	472/1832 (25·8%)	..
	Completed secondary, tertiary, or further school	3408/11 107 (30·7%)	993/2295 (43·3%)	150/1435 (10·5%)	564/1313 (43·0%)	270/1344 (20·1%)	186/1519 (12·2%)	836/1369 (61·1%)	409/1832 (22·3%)	..
Marital status										<0·0001
	Married	5656/11 105 (50·9%)	1019/2294 (44·4%)	451/1433 (31·5%)	777/1316 (59·0%)	681/1341 (50·8%)	798/1520 (52·5%)	717/1369 (52·4%)	1213/1832 (66·21%)	..
	Unmarried	5449/11 105 (49·1%)	1275/2294 (55·6%)	982/1433 (68·5%)	539/1316 (41·0%)	660/1341 (49·2%)	722/1520 (47·5%)	652/1369 (47·6%)	619/1832 (33·8%)	..
Lifestyle factors at baseline*
Social engagement										<0·0001
	Low	1492/11 143 (13·4%)	330/2300 (14·3%)	173/1441 (12·0%)	92/1323 (7·0%)	150/1353 (11·0%)	236/1521 (15·5%)	225/1373 (16·4%)	286/1832 (15·6%)	..
	Low to medium	3576/11 143 (32·1%)	741/2300 (32·2%)	432/1441 (30·0%)	278/1323 (21·0%)	359/1353 (26·5%)	455/1521 (29·9%)	373/1373 (27·2%)	938/1832 (51·2%)	..
	Medium to high	3667/11 143 (32·9%)	784/2300 (34·1%)	544/1441 (37·8%)	448/1323 (33·9%)	468/1353 (34·6%)	507/1521 (33·3%)	384/1373 (28·0%)	532/1832 (29·0%)	..
	High	2408/11 143 (21·6%)	445/2300 (19·3%)	292/1441 (20·3%)	505/1323 (38·2%)	376/1353 (27·8%)	323/1521 (21·2%)	391/1373 (28·5%)	76/1832 (4·1%)	..
Smoking										<0·0001
	Never	7140/11 100 (64·3%)	1229/2294 (53·6%)	751/1439 (52·2%)	1095/1319 (83·0%)	729/1326 (55·0%)	1039/1521 (68·3%)	999/1369 (73·0%)	1298/1832 (70·9%)	..
	Ever	2478/11 100 (22·3%)	614/2294 (26·8%)	508/1439 (35·3%)	177/1319 (13·4%)	445/1326 (33·6%)	345/1521 (22·7%)	298/1369 (21·8%)	91/1832 (5·0%)	..
	Current	1482/11 100 (13·4%)	451/2294 (19·7%)	180/1439 (12·5%)	47/1319 (3·6%)	152/1326 (11·5%)	137/1521 (9·0%)	72/1369 (5·3%)	443/1832 (24·2%)	..
Light or moderate alcohol intake		1004/10 486 (9·6%)	201/2267 (8·9%)	81/1431 (5·7%)	59/1289 (4·6%)	207/791 (26·2%)	258/1509 (17·1%)	129/1367 (9·4%)	69/1832 (3·8%)	<0·0001
Fish intake										<0·0001
	Never	1596/11 095 (14·4%)	198/2294 (8·6%)	480/1430 (33·6%)	102/1320 (7·7%)	57/1333 (4·3%)	414/1517 (27·3%)	307/1369 (22·4%)	38/1832 (2·1%)	..
	Some days	7547/11 095 (68·0%)	1859/2294 (81·0%)	834/1430 (58·3%)	959/1320 (72·7%)	596/1333 (44·7%)	1027/1517 (67·7%)	1038/1369 (75·8%)	1234/1832 (67·4%)	..
	Most days	1843/11 095 (16·6%)	229/2294 (10·0%)	110/1430 (7·7%)	224/1320 (17·0%)	646/1333 (48·5%)	74/1517 (4·9%)	24/1369 (1·8%)	536/1832 (29·3%)	..
	Every day	109/11 095 (1·0%)	8/2294 (0·3%)	6/1430 (0·4%)	35/1320 (2·7%)	34/1333 (2·6%)	2/1517 (0·1%)	0/1369	24/1832 (1·3%)	..
Physically active										<0·0001
	Not (very)	3736/11 090 (33·7%)	636/2291 (27·8%)	482/1434 (33·6%)	365/1317 (27·7%)	444/1333 (33·3%)	502/1514 (33·2%)	412/1369 (30·1%)	895/1832 (48·9%)	..
	Fairly	4681/11 090 (42·2%)	1030/2291 (45·0%)	422/1434 (29·4%)	563/1317 (42·7%)	642/1333 (48·2%)	684/1514 (45·2%)	708/1369 (51·7%)	632/1832 (34·5%)	..
	Very	2673/11 090 (24·1%)	625/2291 (27·3%)	530/1434 (37·0%)	389/1317 (29·5%)	247/1333 (18·5%)	328/1514 (21·7%)	249/1369 (18·2%)	305/1832 (16·6%)	..
Health-related factors at baseline*
Obesity (waist-to-height ratio ≥0·63 [male] or ≥0·58 [female])		4197/10 493 (40·0%)	739/2267 (32·6%)	606/1416 (42·8%)	568/1281 (44·3%)	468/1063 (44·0%)	789/1473 (53·6%)	572/1170 (48·9%)	455/1823 (25·0%)	<0·0001
High total cholesterol		785/6962 (11·3%)	294/1838 (16·0%)	150/1086 (13·8%)	42/558 (7·5%)	159/942 (16·9%)	69/1329 (5·2%)	71/1209 (5·9%)	NA	<0·0001
High systolic blood pressure		3301/10 681 (30·9%)	1229/2296 (53·5%)	416/1430 (29·1%)	74/1321 (5·6%)	379/1096 (34·6%)	349/1518 (23·0%)	277/1192 (23·2%)	577/1828 (31·6%)	<0·0001
Diabetes		1926/11 108 (17·3%)	429/2289 (18·7%)	215/1437 (15·0%)	127/1317 (9·6%)	198/1345 (14·7%)	346/1520 (22·8%)	445/1369 (32·5%)	166/1831 (9·1%)	<0·0001
Stroke		678/11 111 (6·1%)	151/2294 (6·6%)	99/1438 (6·9%)	83/1317 (6·3%)	81/1341 (6·0%)	98/1521 (6·4%)	94/1368 (6·9%)	72/1832 (3·9%)	0·0040
Traumatic brain injury		965/11 092 (8·7%)	125/2290 (5·5%)	151/1439 (10·5%)	184/1316 (14·0%)	194/1325 (14·6%)	237/1521 (15·6%)	65/1369 (4·7%)	9/1832 (0·5%)	<0·0001
Depressive symptoms		1758/11 143 (15·8%)	289/2300 (12·6%)	349/1441 (24·2%)	253/1323 (19·1%)	223/1353 (16·5%)	381/1521 (25·0%)	252/1373 (18·4%)	11/1832 (0·6%)	<0·0001
Physical function at baseline*
Needs help with handling money		323/11 106 (2·9%)	63/2275 (2·8%)	82/1441 (5·7%)	40/1321 (3·0%)	52/1352 (3·8%)	47/1519 (3·1%)	24/1372 (1·7%)	15/1826 (0·8%)	<0·0001
One or more reported difficulties with ADL or IADL		1950/11 143 (17·5%)	320/2300 (13·9%)	388/1441 (26·9%)	162/1323 (12·2%)	280/1353 (20·7%)	392/1521 (25·8%)	280/1373 (20·4%)	128/1832 (7·0%)	<0·0001
Cognition at baseline*
Subjective memory impairment (GMS score)										<0·0001
	0	4430/11 110 (39·9%)	790/2296 (34·4%)	440/1441 (30·5%)	360/1319 (27·3%)	517/1346 (38·4%)	546/1518 (36·0%)	486/1370 (35·5%)	1291/1820 (70·9%)	..
	1–3	5515/11 110 (49·6%)	1227/2296 (53·4%)	791/1441 (54·9%)	795/1319 (60·3%)	626/1346 (46·5%)	808/1518 (53·2%)	777/1370 (56·7%)	491/1820 (27·0%)	..
	>3	1165/11 110 (10·5%)	279/2296 (12·2%)	210/1441 (14·6%)	164/1319 (12·4%)	203/1346 (15·1%)	164/1518 (10·8%)	107/1370 (7·8%)	38/1820 (2·1%)	..
Verbal fluency (<18 animals)		7125/11 143 (63·9%)	1377/2300 (59·9%)	1155/1441 (80·2%)	755/1323 (57·1%)	644/1353 (47·6%)	1109/1521 (72·9%)	1082/1373 (78·8%)	1003/1832 (54·7%)	<0·0001
Delayed recall										<0·0001
	7–10 words	2244/11 143 (20·1%)	519/2300 (22·6%)	160/1441 (11·1%)	237/1323 (17·9%)	334/1353 (24·7%)	188/1521 (12·4%)	221/1373 (16·1%)	585/1832 (31·9%)	..
	5–6 words	3870/11 143 (34·7%)	936/2300 (40·7%)	485/1441 (33·7%)	466/1323 (35·2%)	514/1353 (38·0%)	511/1521 (33·6%)	450/1373 (32·8%)	508/1832 (27·7%)	..
	0–4 words	5029/11 143 (45·1%)	845/2300 (36·7%)	796/1441 (55·2%)	620/1323 (46·9%)	505/1353 (37·3%)	822/1521 (54·0%)	702/1373 (51·1%)	739/1832 (40·3%)	..
Clinical dementia rating24
	0 (no dementia)	6749/11 143 (60·6%)	1468/2300 (63·8%)	693/1441 (48·1%)	919/1323 (69·5%)	749/1353 (55·4%)	775/1521 (51·0%)	854/1373 (62·2%)	1291/1832 (70·5%)	<0·0001
	0·5	4190/11 143 (37·6%)	800/2300 (34·8%)	688/1441 (47·7%)	375/1323 (28·3%)	580/1353 (42·9%)	713/1521 (46·9%)	504/1373 (36·7%)	530/1832 (29·0%)	..
	1 (mild dementia)	200/11 143 (1·8%)	31/2300 (1·3%)	57/1441 (4·0%)	29/1323 (2·2%)	24/1353 (1·8%)	33/1521 (2·2%)	15/1373 (1·1%)	11/1832 (0·6%)	..
	2 (moderate dementia)	4/11 143 (<0·1%)	1/2300 (<0·1%)	3/1441 (0·2%)	0/1323	0/1353	0/1521	0/1373	0/1832	..

Data are n/N (%), where N is the number of participants with non-missing data, or mean (SD). ADL=activities of daily living. GMS=Geriatric Mental State examination. IADL=instrumental activities of daily living. NA=not available.

Baseline characteristics were calculated for the participants without baseline dementia.

Participants were re-interviewed at a mean of 3·8 years (SD 1·2; range 1 month to 7·4 years) follow-up. The number of incident dementia cases in the total sample was 1069 (incidence rate 24·9 cases per 1000 person-years), with higher rates in China (207 cases; 25·3 per 1000 person-years), the Dominican Republic (165 cases; 26·3 per 1000 person-years), Venezuela (155 cases; 29·0 per 1000 person-years), Puerto Rico (153 cases; 27·4 per 1000 person-years), and Mexico (130 cases; 30·5 per 1000 person-years), and lower rates in Cuba (182 cases; 19·5 per 1000 person-years) and Peru (77 cases; 19·3 per 1000 person-years).

With the exception of cholesterol, the proportion of missing data was low for all variables (ranging from 0·1% for age to 5·9% for alcohol; appendix 3 p 6). Data on cholesterol, which is needed to calculate the CAIDE and ANU-ADRI scores, was missing in 4181 (37·5%) individuals, largely because blood samples were not collected in China.

The discriminative performance of each of the five models when mapped in each country is shown in figure 1 (see appendix 3 [p 7] for the full results). Compared with the discriminative performance in the development cohorts, discrimination performance was poor with the CAIDE (0·52≤c≤0·63 across study sites vs c=0·77 in the development cohort) and AgeCoDe (0·57≤c≤0·74 vs c=0·84) models, particularly when mapped in the Dominican Republic and China. By contrast, discrimination of the BDSI (0·62≤c≤0·78 across study sites vs 0·68≤c≤0·78 in the development cohorts), ANU-ADRI (0·66≤c≤0·78 vs 0·65≤c≤0·73), and BDRM (0·66≤c≤0·78 vs 0·75≤c≤0·78) tended to be at a similar level to that reported in the development cohorts in HICs. Across countries, the models consistently worked better at predicting risk of dementia in Peru and worst in the Dominican Republic and China. Figure 2 shows the three models with the highest predictive accuracy for dementia in each of the 10/66 Study sites.

Figure 1

Comparison of the predictive accuracy of each model when mapped in the 10/66 Study compared with the development cohort(s)

c-statistics for the original studies are shown in green. The dashed line indicates the cutoff value for acceptable predictive accuracy (c-statistic ≥0·70). AgeCoDe=Study on Aging, Cognition and Dementia. ANU-ADRI=Australian National University Alzheimer's Disease Risk Index. BDRM=Rotterdam Study Basic Dementia Risk Model. BDSI=Brief Dementia Screening Indicator. CAIDE=Cardiovascular Risk Factors, Aging and Dementia risk score. CHS=Cardiovascular Health Study. CVHS=Cardiovascular Health Cognition Study. c=concordance. EPOZ=Epidemiological Prevention Study of Zoetermeer. FHS=Framingham Heart Study. HRS=Health and Retirement Survey. KP=Kungsholmen Project. MAP=Rush Memory and Aging Project. SALSA=The Sacramento Area Latino Study on Aging.

Figure 2

Discrimination accuracy of models by study site

The three models with the highest accuracy at each site, and their respective c-statistics, are shown. AgeCoDe=Study on Aging, Cognition and Dementia. ANU-ADRI=Australian National University Alzheimer's Disease Risk Index. BDRM=Rotterdam Study Basic Dementia Risk Model. BDSI=Brief Dementia Screening Indicator. c=concordance.

In terms of calibration, using the GB test, all models showed good calibration (p>0·05), with the exception of the ANU-ADRI (χ2=21·4, p=0·011) and BDSI (χ2=18·9, p=0·026) when mapped in Puerto Rico and the BDRM when mapped in all countries, where the results of the GB test suggested that fit was poor. The Cox-Snell residual plots for each risk model when mapped across the different countries are shown in the appendix 3 (pp 14–18). Models generally showed a good fit at low and intermediate risk. However, at high risk, predicted values were extreme, either leading to underestimation or overestimation in most models, particularly for the BDRM and BDSI.

In a meta-analysis, the pooled results for each model showed similar predictive accuracy when mapped in the individual countries compared with the total sample (figure 3). However, the results suggested large heterogeneity across countries for the AgeCoDe model (I2=80·5%) and BDRM (I2=68·9%). Heterogeneity in the other models was low (I2 range 0·0–51·1%; figure 3).

Figure 3

Predictive accuracies of each model according to random-effects meta-analyses

AgeCoDe=Study on Aging, Cognition and Dementia. ANU-ADRI=Australian National University Alzheimer's Disease Risk Index. BDRM=Rotterdam Study Basic Dementia Risk Model. BDSI=Brief Dementia Screening Indicator. CAIDE=Cardiovascular Risk Factors, Aging, and Dementia risk score. c=concordance.

In sensitivity analyses, we found no significant differences between the sociodemographic or health characteristics of individuals included in the analysis and those recruited but excluded from the analysis. In addition, defining the time of diagnosis as the midway point between the baseline and follow-up visits did not affect the outcomes (data not shown).

Discussion

In this study, we externally validated five dementia risk prediction models, developed and tested in HICs, in seven LMICs using data from the 10/66 Study from sites in China, Cuba, the Dominican Republic, Mexico, Peru, Puerto Rico, and Venezuela. The results indicate that while some models (ANU-ADRI, BDSI, and BDRM) transported well from HICs to LMICs, others (CAIDE and AgeCoDe) did not. Results also varied by country, with the highest predictive accuracy reported in Peru and the lowest in China and the Dominican Republic. Given the increasing interest in dementia prevention, particularly in LMICs where resources are limited, it is essential to develop accurate and valid methods for successful prediction of dementia risk to ensure that the right people are targeted for intervention.

The CAIDE and AgeCoDe models did not replicate well when mapped in the different countries from the 10/66 Study. With regard to the CAIDE model, this poor replication is probably due to methodological and sample characteristic differences between the original development study and the 10/66 Study (see appendix 3 [p 8] for a comparison of the sample characteristics across the CAIDE and 10/66 studies). Indeed, the CAIDE score was designed to assess mid-life risk (model development sample age range 39–64 years), with prediction over a mean follow-up period of 20·9 years (SD 4·9). By contrast, the 10/66 Study sample was older (age range 65–106 years) with a mean follow-up of 3·8 years (1·2). The results here support previous external validation studies of the CAIDE score that have shown good transportability within a middle-aged cohort, and poor transportability when applied in cohorts of older participants.9, 17

With regard to the AgeCoDe model, although the follow-up periods were similar between the original study and our study, the AgeCoDe sample was older (all participants were aged ≥75 years) and had higher educational attainment overall than the 10/66 Study sample (see appendix 3 [p 9] for a comparison of the sample characteristics across the AgeCoDe and 10/66 studies). Poor prediction using the AgeCoDe score could therefore be because the associations between risk factors or protective factors and incident dementia are age-dependent and education-dependent, such that the importance of the different risk and protective factors depends on the time of testing and the sample characteristics (ie, HIC vs LMIC). The AgeCoDe score incorporates demographic (age), cognitive function (subjective memory impairment, memory function, and global cognitive function), and physical status (impairments in instrumental activities of daily living) variables. Although these variables might be important for predicting dementia in very late life, the model fails to capture health (ie, cardiometabolic and cerebrovascular), lifestyle (ie, smoking, diet, and alcohol use), and sociodemographic (ie, deprivation) factors that have been found to be important predictors of dementia in mid-life to early-late life and in LMICs. Furthermore, given that cognitive variables were the main predictors in the models, the poor predictive performance might have been because the 10/66 Study sample was characterised by low educational attainment, meaning that cognitive test scores might not be as good at discriminating risk in this setting because of differing effects.27, 28

In contrast to the CAIDE and AgeCoDe models, the ANU-ADRI model (which incorporated age, sex, education, waist-to-height ratio, diabetes, depression, cholesterol level, traumatic brain injury, smoking, alcohol, social engagement, physical activity, and fish intake) and the BDSI model (which incorporated, education, waist-to-height ratio, diabetes, stroke, physical function or impairment in instrumental activities of daily living, and depression) transported well to LMICs, showing similar levels of accuracy to that seen in HICs (see appendix 3 [pp 10–12] for a comparison of the sample characteristics across the 10/66 Study and studies used for the initial development and testing of the ANU-ADRI and BDSI). These models were also found to be well calibrated. Overall, these models include similar predictors such as demographics, mental health, and cardiometabolic status, suggesting that, as in HICs, these factors add relevant information for dementia prediction in LMICs (table 3).

Table 3

Predictor variables included in the dementia risk prediction models

	AgeCoDE	CAIDE	BDSI	ANU-ADRI	BDRM
Age	Yes	Yes	Yes	Yes	Yes
Sex	..	Yes	..	Yes	..
Subjective memory impairment	Yes	..	..	..	Yes
Verbal fluency	Yes	..	..	..	..
Delayed recall	Yes	..	..	..	..
Mini-Mental State Examination	Yes	..	..	..	..
Instrumental activities of daily living	Yes	..	..	..	..
Education	..	Yes	Yes	Yes	..
Systolic blood pressure	..	Yes	..	..	..
Body-mass index	..	Yes	Yes	Yes	..
Cholesterol	..	Yes	..	Yes	..
Physical activity	..	Yes	..	Yes	..
Diabetes	..	..	Yes	Yes	..
Stroke	..	..	Yes	..	Yes
Needs help with medication or handling money	..	..	Yes	..	Yes
Depressive symptoms	..	..	Yes	Yes	..
Traumatic brain injury	..	..	..	Yes	..
Cognitive stimulating activities	..	..	..	Yes	..
Social network	..	..	..	Yes	..
Smoking	..	..	..	Yes	..
Alcohol	..	..	..	Yes	..
Fish intake	..	..	..	Yes	..
Pesticide exposure	..	..	..	Yes	..

Predictor variables can be measured differently (eg, education is categorical [0–6, 7–9, and ≥10 years] in CAIDE but binary [<12 and ≥12 years] in BDSI). AgeCoDe=Study on Aging, Cognition and Dementia. ANU-ADRI=Australian National University Alzheimer's Disease Risk Index. BDRM=Rotterdam Study Basic Dementia Risk Model. BDSI=Brief Dementia Screening Indicator. CAIDE=Cardiovascular Risk Factors, Aging, and Dementia Risk Score.

Regarding the BDRM results, the discriminant accuracy of this model was in an acceptable range (c-statistic ≥0·70) and similar to that reported in the original development study. Compared to the other models tested, the BDRM has fewer predictors incorporating age, subjective memory impairment, history of stroke, and interference with finances and medications. Despite having a small number of predictors, this model could be performing well because it uses the best possible combination of risk variables. However, despite reasonable discriminative accuracy, the GB calibration test showed that the model did not fit well across the different countries (see appendix 3 [p 13] for a comparison of the sample characteristics across the Rotterdam Study, Epidemiological Prevention Study of Zoetermeer, and 10/66 Study). Poor calibration could be due to the small sample sizes in some of the risk quantiles; the GB calibration test uses deciles to create risk groups and we found fewer than five individuals for some risk quantiles. Modification of the group size could improve calibration. Furthermore, the results from the calibration plots indicate that, although the predicted risks at low and intermediate levels were acceptable, like the other models, they were particularly miscalibrated at high risk levels. Finally, the BDRM was designed to predict 10-year risk of dementia, whereas in the 10/66 Study prediction was over a much shorter period.

Overall, the results suggest that additional work is needed to identify the best combination of predictive variables and the optimum scores for each variable to increase the accuracy of predicting dementia in LMICs. As a start, the ANU-ADRI, BDSI, and BDRM models show promise. These models have a number of strengths. First, they incorporate variables that are easy to measure and do not require specialised training to collect. Second, the small number of included predictors enhances their clinical feasibility and potential uptake in LMIC settings.

The models did not perform equally well across countries, with large differences in predictive accuracy across the different sites. Overall, the models had the highest accuracy for predicting dementia in Peru and lowest in China and the Dominican Republic, suggesting that the predictor variables (and their combination to produce risk scores) incorporated in the different models are not robust indicators of future dementia risk across all LMIC settings. It is difficult to determine what is driving the inter-country differences. Although all sites were in LMICs, the countries differ culturally and have different disease risk profiles, age profiles, mortality rates, life expectancies, views on health and ageing, and political and economic systems (including health spending). Even across countries such as Peru and the Dominican Republic, there are considerable differences (eg, in health systems, life expectancy, and education levels). Indeed, the Dominican Republic has the lowest levels of education across all sites, which might explain some of the differences in the results. Furthermore, models might not transport well across settings because of the unique risk factors for dementia (including low educational attainment, socioeconomic status, ethnicity, and poverty) in LMICs compared with HICs, which need to be considered when determining risk in LMICs. The findings could indicate that the models are country-specific and need to be recalibrated to the new setting. Indeed, the combination of risk factors in each model might be appropriate, but the weighting (or risk score) assigned to each factor might need to be adjusted across the different countries. For instance, in the BDSI, a score of 6 is assigned to history of stroke, but in LMICs the score might need to be greater (or less) than 6. Alternatively, it could be that the unique combination of factors needs to be adjusted by site (eg, smoking is common in China and might not be as sensitive for discriminating high-risk vs low-risk cases compared with other sites where smoking prevalence is lower). However, recalibration and development of new models is beyond the scope of this Article, in which the focus is external validation.

The strengths of this study include the use of a large sample with very few missing data (apart from cholesterol data in China) from seven different LMICs, which made it possible to test multiple prediction models in different countries using the same research methods. There were, however, some limitations. First, some adaptations were made to the models to enable their calculation in the 10/66 Study. For example, BMI (required for mapping obesity in the CAIDE model and underweight in the BDSI model) was missing because weight was not measured in the baseline assessment. Instead we used waist-to-height ratio, which, despite having been found to be a better measure of central adiposity and more strongly associated with impaired cardiometabolic health, meant that the comparisons between studies were not identical. In addition, the CAIDE model requires knowledge of mid-life health status, which was not available in the 10/66 Study. Hence, a more detailed assessment of the CAIDE model in LMIC settings using information on mid-life risk factors is needed. Second, because of restrictions in data availability, we were able to test only a limited number of models. As a result, we could have missed identifying a model that might have validated well, with higher predictive accuracy than observed here. Third, our study was restricted to LMICs in South America, China, and the Caribbean. Given the inter-country differences in model performance, the results might therefore not generalise to all LMICs, including, for example, LMICs in South Asia, Africa, and the Middle East. Further work is needed to extend the analysis to other LMICs. Finally, having done a complete-case analysis might have biased the results by limiting their generalisability; however, a sensitivity analysis found no differences in the socioeconomic characteristics or health status between those included and excluded from the study.

As populations continue to age rapidly, the number of people with dementia is predicted to rise, with some of the biggest increases expected to be seen in LMICs. In the absence of any curative treatment for dementia, prevention and the proactive management of modifiable risk factors to delay or slow the onset or progression of the disease are key action areas in the WHO Global Action Dementia Plan. Our findings highlight that the ANU-ADRI, BDSI, and BDRM models could be used in LMICs to help identify individuals for frequent monitoring and targeted dementia risk reduction. However, additional work is needed to test these models in other LMICs before recommendations can be made regarding the best model for use in LMIC settings; and to determine whether model refinement enhances dementia risk prediction in LMICs.

Data sharing

The data underlying this study are restricted as participants did not consent to sharing their information publicly. Data are freely available from the 10/66 Dementia Research Group public data archive for researchers who meet the criteria for access to confidential data. Information on procedures to apply for access to data is available on the 10/66 Dementia Research Group website, or by contacting Prof Martin Prince at dementiaresearchgroup1066@kcl.ac.uk.