How well can electronic health records from primary care identify Alzheimer's disease cases?

BACKGROUND: Electronic health records (EHR) from primary care are emerging in Alzheimer's disease (AD) research, but their accuracy is a concern. We aimed to validate AD diagnoses from primary care using additional information provided by general practitioners (GPs), and a register of dementias. PATIENTS AND METHODS: This retrospective observational study obtained data from the System for the Development of Research in Primary Care (SIDIAP). Three algorithms combined International Statistical Classification of Diseases (ICD-10) and Anatomical Therapeutic Chemical codes to identify AD cases in SIDIAP. GPs evaluated dementia diagnoses by means of an online survey. We linked data from the Register of Dementias of Girona and from SIDIAP. We estimated the positive predictive value (PPV) and sensitivity and provided results stratified by age, sex and severity.
RESULTS: Using survey data from the GPs, PPV of AD diagnosis was 89.8% (95% CI: 84.7-94.9). Using the dataset linkage, PPV was 74.8 (95% CI: 73.1-76.4) for algorithm A1 (AD diagnoses), and 72.3 (95% CI: 70.7-73.9) for algorithm A3 (diagnosed or treated patients without previous conditions); sensitivity was 71.4 (95% CI: 69.6-73.0) and 83.3 (95% CI: 81.8-84.6) for algorithms A1 (AD diagnoses) and A3, respectively. Stratified results did not differ by age, but PPV and sensitivity estimates decreased amongst men and severe patients, respectively.
CONCLUSIONS: PPV estimates differed depending on the gold standard. The development of algorithms integrating diagnoses and treatment of dementia improved the AD case ascertainment. PPV and sensitivity estimates were high and indicated that AD codes recorded in a large primary care database were sufficiently accurate for research purposes.

Introduction

Alzheimer’s disease (AD) entails a heavy burden for patients, their families and public health systems. Its prevalence and economic costs are forecasted to increase dramatically in coming decades due to worldwide population aging.1,2 Electronic health records (EHR) might be useful to update the AD epidemiology, especially in primary care settings. Paradoxically, although general practitioners (GPs) are the gatekeepers of health care services and thus play a pivotal role in the recognition of AD, primary care EHR have scarcely been applied in dementia studies, and even less frequently in AD studies.3,4

A concern about using EHR is the accuracy of diagnoses, as reported in research on dementia diagnoses in primary care,5 but recent studies report high positive predictive value (PPV), indicating that dementia codes in primary care databases were sufficiently accurate for research use.3,6 Nonetheless, literature about the accuracy of EHR in AD is mainly focused on hospital records7–9 and data from primary care is scant.4,10

We hypothesized that EHR from primary care would be a valid tool to study AD, since dementia diagnoses are accurate6 and AD is the most prevalent dementia subtype. The present study sought to validate AD diagnoses recorded in a primary care database by comparing AD from primary care with information from an online survey of GPs and with data from a registry of dementia diagnoses.

Materials and methods

We conducted a retrospective observational study, approved by the Clinical Research Ethics Committee of the Primary Care Research Institute IDIAP Jordi Gol (IDIAPJGol). We followed the STARDS 2015 guidelines11 and the RECORD statement.12

We used data from the Information System for Research in Primary Care (SIDIAP database), which contains anonymized longitudinal medical records related to demographics including deprivation index,13 symptoms, diagnoses, laboratory tests, prescriptions, and pharmacy dispensing from about 6 million patients (>80% of the Catalan population) attended in the primary care settings of the Catalan Health Service.14,15 Accuracy of SIDIAP data have been analyzed for several conditions16–18 including overall dementia,6 but not yet for AD.

Using SIDIAP data, we identified AD cases using algorithms that combined EHR, a method previously applied to identify dementia cases.10,19 We followed Imfeld et al 2013 to define three algorithms that combine information about diagnoses and pharmacological treatment to identify AD cases (Table 1). We considered treated patients as cases because in Catalonia the prescription of anti-dementia drugs can be requested by the GP but requires approval from a geriatrician, a psychiatrist, or a neurologist.

Table 1

Algorithms to identify Alzheimer's disease (AD) cases in electronic health records from primary care, using codes from the international statistical classification of diseases and related health problems, version 10 (ICD-10) and from the Anatomical Therapeutic Chemical (ATC)

Algorithm	Definition of AD case
A1	Diagnosed patients: have an ICD10 code for AD (F00 or G30).
A2	Diagnosed or treated patients: have a code for AD (ICD10: F00 or G30) or for prescription or billing of anti-dementia drugs (ATC: N06DA, N06DX01).
A3	Diagnosed or treated patients without previous conditions: have a code for AD (ICD10: F00 or G30) or for prescription or billing of anti-dementia drugs (ATC: N06DA, N06DX01).Treated patients were included if they had no code of dementia diagnosis or had a code of unspecified dementia (F03), and were excluded if they had a code for: a specific subtype of dementia such as Lewy bodies dementia, vascular or frontotemporal dementia (ICD10: F01, F02); Parkinson (ICD10: G20-G22); anti-Parkinson drugs (ATC: N04); or cerebrovascular disease (ICD10:I60- I69, G45, G46) within two years prior to AD diagnosis.

We followed two approaches. First, we asked the GPs to complete an online survey to provide additional information about dementia diagnoses.6 Our previously published validation study focused on diagnoses of overall dementia recorded in SIDIAP; here, we provide a sub-analysis restricted to AD diagnoses (International Statistical Classification of Diseases, 10th revision (ICD-10): F00, G30). Second, we linked SIDIAP data with a dementia-specific registry for the province of Girona.

Survey sent to GPs

Survey questions, design and administration are described elsewhere.6 Briefly, GPs from the IDIAPJGol Agency of Clinical Research Management in Primary Care (AGICAP) network20 were invited to participate. Those who agreed to participate were sent the online survey in January 2018 and given 1 month to evaluate a consecutive series of their patients with a dementia diagnosis recorded in the EHR.6 The survey asked about the current basis for a dementia diagnosis, the fulfillment of DSM-IV or ICD-10 diagnosis criteria, and the dementia subtype (Figure S1). Patients were defined as AD cases if they were identified as a true dementia case (diagnosed by a specialist; or based on cognitive and functional tests or a clinical opinion) and the GP confirmed the dementia subtype as AD. Such confirmation of dementia subtype was based on the fulfillment of the DSM-IV or the ICD10 diagnostic criteria. The inclusion of both sets of diagnostic criteria aimed to facilitate the identification of true cases, because some physicians might be more familiarized with one of them.

Linkage between SIDIAP and the registry of dementias

The Registry of Dementias of Girona (ReDeGi) contains demographic and clinical data of all the incident cases of dementia diagnosed in the seven hospitals of the public health care system within the province of Girona.21 The ReDeGi uses standardized criteria for case definition, and follows the Center for Disease Control and Prevention guidelines on surveillance systems.22 The ReDeGi includes the Clinical Dementia Rating score,23 a measure of disease severity (mild, moderate and severe). A recent study found high adherence to the clinical practice guidelines among physicians in memory clinics in the ReDeGi catchment area.24

We included patients recorded in ReDeGi between 2007 and 2016, and ascribed to primary care settings of the Catalan Health Service.The linkage was carried out using the national health identifier from the Catalan Health Service, which was available in both databases. We encoded the national health identifier and we used this ciphered identifier in all linkage procedures (Figure 1). Researchers did not have access to the national health identifier or any other kind of identifying data.

Figure 1

Register of dementias of Girona (ReDeGi) and SIDIAP linkage flow diagram. Data confidentiality measures were applied during information transfer (encoded data in grey arrows; password in dotted grey lines).Abbreviation: SIDIAP, Information System for Research in Primary Care.

In ReDeGi, AD cases were identified according to ICD-10 codes, excluding mixed dementia cases. In SIDIAP, we identified AD cases using the above-mentioned algorithms combining EHR (Table 1).

Statistical analysis

We used percentages to describe categorical variables and mean (SD) for continuous variables. Using data from the survey of GPs, we calculated the PPV including the number of confirmed diagnoses of AD in the numerator, against all the diagnoses of AD evaluated by GP, in the denominator. We replicated the PPV estimate considering AD diagnoses evaluated and not evaluated by GPs, as sensitivity analysis: we included AD diagnoses confirmed by the GPs or with evidence of treatment with anti-dementia drugs in the numerator, and all AD diagnoses (evaluated or not by GPs) in the denominator. Using data from the linkage between SIDIAP and ReDeGi, we estimated the PPV and sensitivity of AD cases. We calculated the PPV considering the number of AD cases identified in both ReDeGi and SIDIAP in the numerator, and the AD cases identified in SIDIAP, in the denominator. Sensitivity was defined as the number of AD cases identified in both ReDeGi and SIDIAP in the numerator, and the AD cases identified in ReDeGi in the denominator. For both PPV and sensitivity, results were stratified by AD definition (Table 1), age, sex and severity. We provided 95% CI assuming a normal distribution for all estimates. The level of significance was defined as p-value=0.05. All analyses were performed using R software v3.5.2.25

Results

We sent the survey to the 42 GPs who agreed to participate, of which 29 (69%) provided feedback. The quota of patients assigned to these 29 GPs is described in Table S1. Of 188 patients with a diagnosis of AD, 137 were evaluated within the 1-month response period (Figure 2). Table 2 provides a description of patient characteristics.

Table S1

Description of the quota of patients from the 29 general practitioners who answered the survey

Sociodemographic characteristics	Quota of patients (n=29)
Percentage of men, X‾ (sd)	49.5 (3.4)
Age, X‾ (sd)	47.5 (2.0)
Percentage of people aged ≥65 years, X‾ (sd)	21.4 (4.9)
Number of patients, X‾ (sd)	1426 (331)
Rural areas, n (%)	5 (17.1)
Socioeconomic status in urban areas, n (%)
Status 1: less deprived	2 (8.3)
Status 2	2 (8.3)
Status 3	6 (25)
Status 4: more deprived	14 (58.3)

Figure 2

Flow of Alzheimer's disease (AD) diagnoses evaluated by general practitioners (GPs) responding to an online survey.

Table 2

Baseline demographic and clinical characteristics of patients according to the validation method: survey of general practitioners (GPs) to evaluate recorded AD diagnoses or linkage between datasets from primary (SIDIAP) and secondary care (ReDeGi)

	Survey		Dataset
Evaluated patients (n=137)	Not-evaluated patients (n=51)	All selected patients (n=4,966)
Age, X‾ (sd)	80.9 (7.7)	83.5 (6.9)	80.2 (6.9)
Women, n (%)	92 (67%)	36 (70%)	3151 (63%)
Deprivation index, n (%)
Quintile 1: less deprived	12 (9%)	1 (2%)	629 (22.64)
Quintile 2	32 (23%)	14 (27%)	472 (16.99)
Quintile 3	21 (15%)	8 (16%)	545 (19.62)
Quintile 4	34 (25%)	13 (26%)	653 (23.51)
Quintile 5: more deprived	38 (28%)	15 (29%)	479 (17.24)
Antidementia drugs, n (%)	92 (67%)	32 (63%)	3275 (66%)
Severity of dementia, n (%)
Mild	-	-	2955 (60%)
Moderate	-	-	1532 (31%)
Severe	-	-	437 (9%)

Of the diagnoses evaluated by GPs using the survey, 123 were considered as AD cases, and 14 were not confirmed as AD cases (Figure 2). We estimated the PPV in 89.8% (95% CI: 84.7–94.9). The supplementary materials show specific numerators and denominators used to estimate PPVs. Sensitivity analysis considered the 137 AD diagnoses evaluated by GPs using the survey questions and the 51 not evaluated, of which 32 were treated patients. Sensitivity analysis provided a PPV of 82.4 (95%CI: 77.0–87.8) (Supplementary materials). Stratified analyses are shown in Table 3.

Table 3

Positive predictive value (PPV) of Alzheimer's disease (AD) diagnoses recorded in SIDIAP using information provided by the survey of general practitioners as reference. Sensitivity analysis included all AD patients of participating general practitioners (whether their diagnoses were evaluated or not)

Population	n	Main analysis			n	Sensitivity analysis
TP	FP	PPV (95% CI)	TP	FP	PPV (95% CI)
Women	92	84	8	91.3 (85.5–97.1)	128	106	22	82.8 (77.0–87.8)
Men	45	39	6	86.7 (76.7–96.6)	60	49	11	81.6 (71.9–91.4)
<80 years	84	74	10	88.1 (79.9–94.2)	105	89	16	84.8 (77.9–91.6)
≥80 years	54	49	4	92.4 (87.8–100.5)	83	66	17	79.5 (70.8–88.2)
Total	137	123	14	89.8 (84.7–94.9)	188	155	33	82.4 (77.0–87.8)

Abbreviations: TP, true positive; FP, false positive; SIDIAP, Information System for Research in Primary Care.

This linkage provided data for 4,996 patients, as described in Table 2. A specific AD diagnosis was found for 2,728 patients according to ReDeGi, 2,603 in SIDIAP and 1,947 in both databases.

PPV and sensitivity of AD diagnoses were 74.8 (95% CI: 73.1–76.4) and 71.4 (95% CI: 69.6–73.0), respectively (Supplementary materials). When using algorithm A2 (where cases included diagnosed or treated patients), PPV declined and sensitivity increased; when using algorithm A3 (excluding cases with previous conditions), PPV was unchanged and sensitivity increased compared to algorithm A1 (Table 4). No differences were observed when estimates were stratified by age, but men had lower PPV estimates than women, and sensitivity decreased with severity in all AD case definitions (Table 4).

Table 4

Positive predictive value (PPV) and sensitivity of Alzheimer's disease (AD) diagnoses in SIDIAP using three algorithms to define AD cases and linkage with the register of dementias of Girona as reference

Algorithm	Group	Subgroup	TP	FP	FN	PPV (95% CI)	Sensitivity (95% CI)
A1. Diagnosed patients
	Sex	Women	1,365	398	528	77.4 (75.4–79.3)	72.1 (70.0–74.1)
		Men	582	258	253	69.3 (66.1–72.3)	69.3 (66.1–72.3)
	Age	<80 years	843	285	275	74.9 (72.3–77.4)	75.4 (72.8–77.8)
		≥80 years	1,104	371	506	74.8 (72.5–77.0)	68.6 (66.3–70.8)
	Severity	Mild	1346	386	434	77.7 (75.7–79.6)	75.6 (73.6–77.6)
		Moderate	472	220	263	68.2 (64.6–71.6)	64.2 (60.7–67.7)
		Severe	123	43	83	74.1 (66.9–80.2)	59.7 (52.9–66.2)
	Total population		1,947	656	781	74.8 (73.1–76.4)	71.4 (69.6–73.0)
A2. Diagnosed or treated patients
	Sex	Women	1,623	715	270	69.4 (67.5–71.3)	85.7 (84.1–87.2)
		Men	711	484	124	59.5 (56.7–62.2)	85.1 (82.6–87.4)
	Age	<80 years	992	531	126	65.1 (62.7–67.5)	88.7 (86.7–90.4)
		≥80 years	1,342	668	268	66.8 (64.7–68.8)	83.4 (81.4–85.1)
	Severity	Mild	1585	703	195	69.3 (67.4–71.1)	89.0 (87.5–90.4)
		Moderate	590	391	145	60.1 (57.0–63.2)	80.3 (77.2–83.0)
		Severe	152	87	54	63.6 (57.3–69.4)	73.8 (67.4–79.3)
	Total population		2,334	1,199	394	66.1 (64.5–67.6)	85.6 (84.2–86.8)
A3. Diagnosed or treated patients without previous conditions
	Sex	Women	1,599	558	294	74.6 (72.7–76.4)	83.9 (82.2–85.5)
		Men	687	346	148	67.5 (64.5–70.3)	81.8 (79.0–84.3)
	Age	<80 years	973	378	145	72.7 (70.3–75.0)	86.7 (84.6–88.6)
		≥80 years	1,313	527	297	72.1 (69.9–74.1)	80.9 (78.9–82.7)
	Severity	Mild	1552	519	228	74.9 (73.0–76.7)	87.2 (85.6–88.7)
		Moderate	569	280	166	67.0 (63.8–70.1)	77.4 (74.3–80.3)
		Severe	144	59	62	70.9 (64.3–76.7)	69.9 (63.3–75.8)
	Total population		2,286	905	442	72.3 (70.7–73.9)	83.3 (81.8–84.6)

Abbreviations: TP, true positive; FP, false positive; FN, false negative; SIDIAP, Information System for Research in Primary Care.

Discussion

We provided a comprehensive overview of the accuracy of AD diagnoses routinely recorded in primary care. Our PPV and sensitivity estimates indicated high accuracy of AD codes in both global and stratified analyses, similar to the one observed in other conditions such as cancer16 or cardiovascular disease.26,27 We enhanced the robustness of our findings by using two gold standards: a survey to GPs and a linkage with a provincial register of dementia. Estimates differed depending on the gold standard, and values were higher using data from the survey than from the ReDeGi linkage.

When the survey to GPs was the gold standard, our PPV estimate was similar to or slightly higher than previously reported in primary care studies on overall dementia4 or AD.10,28 In other words, patients with an AD diagnosis in EHR were very likely to have the disease. When the linkage with ReDeGi was the gold standard, PPV estimates varied between 66% and 74% depending on the applied algorithm. The lowest PPV was achieved when using algorithm A2, revealing that some treated patients might not actually have AD. The indications for anti-dementia drugs (anticholinesterases and memantine) are relatively specific for AD, but also can be used for dementia with Lewy bodies or other conditions.3 Indeed, when we restricted the treated patients considered as AD (algorithm A3), our PPV improved, likely because the exclusion of patients who had a code indicating other subtypes of dementia, previous cerebrovascular disease or Parkinsonism decreased the false positives. For example, to consider previous cerebrovascular diseases might help discern between AD and vascular dementia, in line with previous EHR-based epidemiological studies of AD prevalence.10

Our PPV estimates differed depending on the gold standard, and were higher when using data from the survey (82–89%) than from the ReDeGi linkage (66–74%). These differences were unlikely to be related with the different study populations (ie, Catalonia in the survey, and Girona province in the linkage), since clinical practices are homogeneous within the primary care services of the Catalan Health Service.16 The variability might be related to differences in validation methods and in diagnostic procedures in primary and secondary care. We cannot discard that selection bias affected PPV values based on the survey, since GPs enrolled voluntarily and not all AD diagnoses were evaluated. However, sensitivity analysis provided similar results, suggesting a low level of overestimation. In parallel, some underestimation of the PPV values based on the linkage may have occurred, because we used a conservative AD definition. Previous studies considered diagnoses of unspecified dementia as AD cases,7 but we restricted this criterion to treated patients, following Imfeld et al (2013). We also used pure AD cases recorded in ReDeGi as a gold standard, while other studies included mixed dementia in the AD case definition.7 In primary care databases, the mixed dementia code is seldom used;10 therefore, if AD codes were used instead, the number of false positives would be increased.

Sensitivity estimates obtained using the linkage analysis showed that more than two-thirds of AD cases had received a diagnosis. Identification of AD cases – treated or not – depends greatly on the GP’s role. Some level of under-recording of AD might occur when GPs use free-text instead of ICD-10 codes to record AD diagnoses, do not discern between dementia subtypes because they consider the diagnosis and treatment of dementia useless,29,30 or identify general dementia in a first visit and specify the subtype in a later visit.7 Our sensitivity estimates improved when patients treated with anti-dementia drugs were included as AD cases (algorithms A2 and A3). Anti-dementia drugs might have been prescribed in primary care without recording a diagnostic code for AD in the EHR, or a diagnosis of AD might have been made in secondary care or in a private clinic, with no record of diagnosis in the primary care database of the Catalan Health Service.31

Our findings disagree with previous evidence suggesting that less than half of the expected number of patients with dementia are recognized in primary care.32 Centers with an active approach to dementia diagnosis are likely to record a higher proportion of community cases on their health care databases.33 Thus, the quality of records regarding subtypes of dementia might substantially depend on the dementia policies implemented in each primary care setting, and their implementation or modifications over time.3 For example, dementia diagnosis rates improved substantially during 10 years in the UK.31 In the Catalan Health Service, dementia is actively considered; eg, a cognitive test is administered in about 25% of people aged at least 65 years – with or without memory complaints.6 Moreover, most of the AD diagnoses recorded in SIDIAP were done by a specialist, suggesting that primary and secondary care are well coordinated in the Catalan Health System.34,35

Finally, we succeeded in identifying AD cases using algorithms. The integration of data related to AD diagnoses, treatment and comorbidities enhanced case identification. We recommend that future EHR studies develop AD case algorithms depending on their research objectives. Large prospective studies might benefit from the definition of AD cases as diagnosed patients to avoid a biased risk estimate due to a large number of false positives.3 In epidemiological studies, such as prevalence estimates, a definition of AD cases as diagnosed or treated patients would ensure identification of the highest number of individuals having the condition.

We acknowledge several limitations. First, the PPV could be overestimated because the validation study was carried out with GPs from the AGICAP network, who are regularly involved in clinical trials and therefore they could tend to be more accurate in registering diagnosis in the electronic history of patients. Second, GPs voluntarily answered the survey, which could affect the representativeness of the study sample. Third, we used ReDeGi as gold standard, but this is a register based on cases recorded in secondary care settings and therefore, it might not capture all AD cases occurred in the Girona province. However, the ReDeGi has proved to cover about 75% of the expected cases of dementia in the area under surveillance.36 Even more, ReDeGi provides good-quality diagnoses made by specialists in neurology, geriatrics, psychiatry or internal medicine from outpatient consultation offices in specialized care or in the hospital memory clinics.

Conclusion

Our findings suggested that EHR from primary care were accurate to identify AD cases, but efforts can be made to improve diagnostic accuracy in men and in severe AD cases. Algorithms combining information on diagnosis and treatment for AD might be a powerful tool in primary care research. Sensitivity estimates differed depending on the method used – survey or linkage with a register,-but were sufficiently high to suggest that the primary care data were accurate for research purposes.