How helpful are the European AIDS Clinical Society cognitive screening questions in predicting cognitive impairment in an aging, well-treated HIV-positive population?

OBJECTIVES: Diagnosing neurocognitive impairment (NCI) in HIV infection requires time-consuming neuropsychological assessment. Screening tools are needed to identify when neuropsychological referral is indicated. We examined the positive and negative predictive values (PPVs and NPVs, respectively) of the three European AIDS Clinical Society (EACS) screening questions in identifying NCI.
METHODS: The Neurocognitive Assessment in the Metabolic and Aging Cohort (NAMACO) study recruited patients aged ≥45 years enrolled in the Swiss HIV Cohort Study between 1 May 2013 and 30 November 2016. NAMACO participants (1) answered EACS screening questions, (2) underwent standardized neuropsychological assessment and (3) completed self-report forms [Center for Epidemiologic Studies Depression Scale (CES-D)] rating mood. NCI categories were defined using Frascati criteria. PPVs and NPVs of the EACS screening questions in identifying NCI categories were calculated.
RESULTS: Of 974 NAMACO participants with complete EACS screening question data, 244 (25.1%) expressed cognitive complaints in answer to at least one EACS screening question, of whom 51.3% had NCI (26.1% HIV-associated and 25.2% related to confounding factors). The PPV and NPV of the EACS screening questions in identifying HIV-associated NCI were 0.35 and 0.7, respectively. Restricting analysis to NCI with functional impairment or related to confounding factors, notably depression, the NPV was 0.90. Expressing cognitive complaints for all three EACS screening questions was significantly associated with depression (P < 0.001).
CONCLUSIONS: The EACS screening questions had an NPV of 0.7 for excluding patients with HIV-associated NCI as defined by Frascati criteria. The PPV and NPV may improve if NCI diagnoses are based on new criteria.

Introduction

In the era of potent antiretroviral therapy (ART), HIV‐associated neurocognitive impairment (NCI) remains a clinical problem, particularly in an aging population of people living with HIV (PLWH). NCI is also a diagnostic problem, as standardized neuropsychological testing of specific cognitive domains is time‐consuming, costly and not available at all centres 1.

Since NCI was first identified as an entity, ART has become more effective and patients now live well and for longer. With this, NCI categories were redefined in 2007, according to the Frascati criteria, into asymptomatic neurocognitive impairment (ANI; mild to moderate cognitive deficits without functional impairment), mild neurocognitive disorders (MND; mild to moderate cognitive deficits with functional impairment) and HIV‐associated dementia (HAD; moderate to severe cognitive deficits with functional impairment) 2. Labelling NCI as ‘HIV‐associated’ requires the exclusion of confounding factors, including organic brain pathology, substance misuse and psychiatric disorders, notably depression 2, 3.

The Frascati criteria are, to date, the only published criteria for categorizing NCI that have been arrived at by consensus. Whilst such criteria enable comparison of the results of cohort studies examining NCI in different patient populations, limitations have been described. Patients with mild cognitive deficits classified as ANI, for example, have been reclassified as cognitively normal when assessed using other criteria 4. Equally, patients at the moderate end of the ANI spectrum may be classified as having ANI rather than MND as a consequence of the low sensitivity of testing methods for functional impairment. Using Frascati criteria alone, it is difficult to predict which individuals with ANI will deteriorate. This is important given the CNS HIV Anti‐Retroviral Therapy Effects Research (CHARTER) study group observation that ANI diagnosis conferred a two‐ to six‐fold increase in the risk of earlier development of symptomatic NCI 5.

Potential NCI screening tools more rapid than neuropsychological testing (minutes rather than hours) have been examined 6, 7, 8, 9, 10. However, as several studies were published prior to the 2007 Frascati criteria 6 or were conducted among younger patients or to identify more severe NCI stages 10 in advanced disease 11, or without excluding patients with depression 7, it is not possible to confidently apply these to aging populations of PLWH who have well‐controlled infection on modern ART.

The European AIDS Clinical Society (EACS) recommends a simple tool to identify which patients merit formal neuropsychological testing, using three cognitive symptom questions which cover memory loss, mental slowing and attention difficulties 12. The questions are taken from a paper by Simioni et al. which assessed patients with cognitive complaints for the presence of NCI 13 and are included in the EACS NCI assessment algorithm at the time of writing 14. In Switzerland, PLWH enrolled in the Swiss HIV Cohort Study (SHCS) 15 are screened for NCI once a year using the EACS screening questions. Recruiting SHCS patients to the Neurocognitive Assessment in the Metabolic and Aging Cohort (NAMACO) study has enabled a review of the value of the EACS screening question scores in identifying NCI in PLWH. The aim of this study was to determine the positive and negative predictive values (PPVs and NPVs, respectively) of these questions.

Methods

Study design

The NAMACO study is an ongoing, prospective, longitudinal, multicentre and multilingual (German, French and Italian) study included within the SHCS, created to investigate NCI in a well‐treated and aging population of PLWH.

SHCS patients aged ≥ 45 years and followed up at one of seven university‐affiliated hospital centres (Bern, Basel, Geneva, Lausanne, St‐Gallen, Lugano and Zurich) were invited to participate in the NAMACO study between 1 May 2013 and 30 November 2016, resulting in a cohort of 981 participants 16. The ethics committees of all participating hospital centres approved the NAMACO study protocol. All NAMACO participants signed informed consent forms prior to inclusion.

Neuropsychological evaluation

NAMACO participants were asked the three EACS screening questions on cognitive function at baseline (inclusion) by their HIV clinicians, as part of a standard SHCS clinic visit: (1) Do you experience frequent memory loss? (2) Do you feel that you are slower when reasoning, planning activities or solving problems? (3) Do you have difficulties paying attention? For each question, the response options were: never, hardly ever or yes, definitely.

All participants then underwent standardized neuropsychological assessment by neuropsychologists, examining seven cognitive domains based on the International Network for Strategic Initiatives in Global HIV Trials (INSIGHT) Strategic Timing of AntiRetroviral Treatment (START) study 17: motor skills, speed of information processing, attention and working memory, executive function, verbal episodic memory, language, and sensory and perceptual skills (Table S1). The complete battery required 90 min to perform in patients with no deficits. The raw score for each neuropsychological test was converted to a demographically adjusted standard score (z‐score) as described elsewhere 16.

Participants also completed self‐report forms on functional ability (Lawton's Instrumental Activities of Daily Living and Patient's Assessment of Own Functioning Inventory questionnaire) and mood [Center for Epidemiologic Studies Depression Scale (CES‐D)]. Functional impairment was defined as difficulties in at least two items out of eleven. Depressive symptoms were considered as mild for CES‐D scores 16–26, and severe for CES‐D scores ≥ 27.

Frascati criteria were used to categorize participants as having: no NCI (normal neuropsychological examination), ANI, MND, HAD or non‐HIV‐associated NCI (‘other’, when NCI could be explained by confounding factors: substance misuse, psychiatric disorders including CES‐D ≥ 27, ART toxicity, central nervous system opportunistic infection, stroke or trauma). The distinction between HIV‐associated NCI and NCI related to confounding factors was based on the clinical judgment of the neuropsychologists performing the neuropsychological assessment. Although the Frascati criteria have limitations as described in the Introduction, they were the only published criteria arrived at by consensus at the time of study recruitment (2013–2016).

Statistical analysis

The association between cognitive complaints and NCI category was examined using the Pearson χ2 test. Cognitive complaints were defined as being present when the patient answered yes, definitely and were analysed as a binary variable (yes, definitely answered to at least one EACS screening question). Predictive values of answering yes, definitely to at least one EACS screening question were assessed in 2 × 2 contingency tables by the ability to detect or exclude (1) all NCI (HIV‐ and non‐HIV‐associated, with and without functional impairment), (2) HIV‐associated NCI and (3) NCI with functional impairment either of HIV origin (MND and HAD) or not (‘other’). Cognitive complaints in relation to depressive symptoms (CES‐D score) were further examined using the Wilcoxon–Mann–Whitney test.

Cognitive complaints were also examined using receiver operating characteristic (ROC) curves by summing the three EACS screening questions (never = 0, hardly ever = 1, and yes, definitely = 2) and taking integer values between 0 and 6 as a pseudo‐continuous means of detecting or excluding NCI, HIV‐associated NCI, NCI with functional impairment and non‐HIV‐associated NCI (other).

Statistical analyses were conducted using r Development Core Team version 3.2 2015 (R Foundation for Statistical Computing, Vienna, Austria; http://www.R-project.org).

Results

The baseline characteristics of the 981 participants enrolled in the NAMACO study have been presented elsewhere 16. Briefly, of the 981 participants, 782 (79.7%) were male, 899 (91.7%) were Caucasian and 627 (63.9%) were aged > 50 years (mean ± SD age 54.5 ± 7.5 years). Most (96.2%) had viral loads < 50 copies/mL; the median nadir CD4 count was 180 cells/μL (IQR 74, 270 cells/μL).

Of the 964 participants (98.3%) with complete NCI data, 127 (13.2%) were categorized as having non‐HIV‐associated NCI (‘other’) (Table 1). Of these participants, several had more than one confounding factor: 79 participants (62.2%) had a psychiatric disorder, mostly depression, 41 (32.3%) had organic brain pathology (history of trauma, stroke, opportunistic infection or unspecified pathology), and 26 (20.5%) had a history of substance misuse.

Table 1

Neurocognitive diagnosis among study patients with cognitive complaints, without cognitive complaints and overall

Neurocognitive diagnosis	Patients with complaints (n = 238)*	Patients without complaints (n = 719)*	All patients (n = 964)*
Normal	116 (48.7)	455 (63.3)	574 (59.5)
ANI	54 (22.7)	193 (26.8)	249 (25.8)
MND	3 (1.3)	5 (0.7)	8 (0.8)
HAD	5 (2.1)	1 (0.1)	6 (0.6)
Other	60 (25.2)	65 (9)	127 (13.2)

Values shown are n (%). ANI, asymptomatic neurocognitive impairment; HAD, HIV‐associated dementia; MND, mild neurocognitive disorder; NCI, neurocognitive impairment; other, neurocognitive impairment related to confounders rather than associated with HIV infection.

This number refers to the number of patients with complete European AIDS Clinical Society screening question data and complete neurocognitive assessment data.

Of the 974 participants with complete EACS screening question data (99.3%), the prevalence of cognitive complaints (answering yes to at least one EACS screening question) was 25.1% (244/974): 21% pertaining to memory loss, 8.3% to mental slowing, and 12.6% to attention deficits. SHCS patients who were eligible for the NAMACO study but not enrolled (n = 2718) presented a lower prevalence of cognitive complaints: 14.4% pertaining to memory loss (P < 0.001), 7% to mental slowing (P = 0.19) and 9.8% to attention deficits (P = 0.02). The neurocognitive diagnoses in these participants, presented according to Frascati criteria, and the presence or absence of neurocognitive complaints are shown in Table 1 and Figure 1. The presence of NCI was significantly associated with cognitive complaints (P < 0.001), with NCI in 122/238 participants (51.3%) with cognitive complaints compared to 264/719 participants (36.7%) without complaints. The PPV and NPV of answering yes, definitely to at least one EACS screening question for diagnosing NCI (HIV‐associated with or without ‘other’ causes) or symptomatic NCI (excluding ANI and normal neurocognitive assessment) are shown in Table 2.

Figure 1

Prevalence of neurocognitive impairment among all study participants, those with cognitive complaints and those without complaints. NCI, neurocognitive impairment; ANI, asymptomatic neurocognitive impairment; MND, mild neurocognitive disorder; HAD, HIV‐associated dementia; other, neurocognitive impairment related to confounders other than those associated with HIV infection.

Table 2

Positive and negative predictive values (PPV and NPV, respectively) of answering ‘yes’ to at least one European Clinical AIDS Society screening question as a screening tool for detecting or excluding neurocognitive impairment

Diagnosis	Question*	Sensitivity	Specificity	PPV	NPV
NCI (ANI/ MND/ HAD/other) versus no NCI (n = 957)	Memory	0.20	0.65	0.42	0.65
	Slowing	0.10	0.89	0.54	0.63
	Concentration	0.13	0.77	0.38	0.62
	Total	0.32	0.80	0.51	0.63
HIV‐associated NCI (ANI/MND/HAD) versus no NCI (n = 932)	Memory	0.19	0.65	0.32	0.70
	Slowing	0.07	0.89	0.36	0.69
	Concentration	0.10	0.77	0.24	0.67
	Total	0.24	0.80	0.35	0.70
Symptomatic NCI (MND/HAD/other) versus asymptomatic (no NCI/ ANI) (n = 957)	Memory	0.22	0.64	0.17	0.91
	Slowing	0.14	0.89	0.29	0.90
	Concentration	0.21	0.79	0.22	0.91
	Total	0.49	0.79	0.29	0.90

ANI, asymptomatic neurocognitive impairment; HAD, HIV‐associated dementia; MND, mild neurocognitive disorder; NCI, neurocognitive impairment; other, neurocognitive impairment related to confounders rather than associated with HIV infection.

The three European AIDS Clinical Society (EACS) questions are shown, pertaining to memory, mental slowing and concentration. The PPV and NPV shown relate to the answer, ‘yes, definitely’ to each of the three questions; ‘Total’ refers to the answer to at least one of the three questions being ‘yes, definitely’.

The presence of cognitive complaints among NAMACO participants was associated with low mood as measured by CES‐D score. Having severe depression (CES‐D score > 27), present in 90/973 participants with complete CES‐D score data (9.2%), was significantly associated with answering yes, definitely to all three EACS screening questions (χ2 test, P < 0.001). However, even patients answering yes, definitely to at least one EACS screening question had higher median CES‐D scores than those having no complaints [median 14 (IQR 7–24) versus median 5 (IQR 4–6); P < 0.0001].

A continuous model, defining complaints in terms of the sum of answers to the three EACS screening questions, gave areas under the ROC curves (AUCs) of 0.57 [95% confidence interval (CI) 0.53–0.61] for distinguishing NCI from no NCI, 0.52 (95% CI 0.47–0.56) for distinguishing HIV‐associated NCI from no NCI, and 0.7 (95% CI 0.64–0.75) for distinguishing MND, HAD or ‘other’ from ANI or no NCI (Figure S1).

Discussion

In this large cohort of patients with well‐controlled HIV infection, one quarter had cognitive complaints, most frequently related to memory loss. The presence of NCI, diagnosed upon formal neuropsychological assessment, was significantly associated with having cognitive complaints. Having cognitive complaints in turn was significantly associated with low mood and depression. PPVs of the EACS screening questions were poor (0.29–0.51) while the NPV to exclude NCI varied between 0.63 and 0.9, depending on NCI category.

This study has several strengths. The NAMACO study, with over 900 participants, is one of the largest cohort studies to examine NCI, and NAMACO participants are highly characterized through links to the SHCS database. Neuropsychological assessment was conducted by trained neuropsychologists to enable inclusion of individuals less able to complete assessments via computer. Assessment was possible despite the study being conducted over three linguistic regions of Switzerland, through neuropsychologists being fluent in the test language, an element vital for the verbal aspect of testing 18. Finally, the EACS screening questions were asked by the patients’ own HIV clinicians, so in a ‘real‐life’ setting rather than as part of neuropsychological testing.

The percentages of participants with cognitive complaints and different categories of NCI differ from those observed by Simioni et al. 13. Although a similar percentage had complaints (25% in the current study compared to 27% reported by Simioni et al.), the percentage of patients with NCI among patients with and without complaints was much higher, with HIV‐associated NCI (ANI, MND and HAD) reported in 84% of complaining patients (52% with MND) and 64% of noncomplaining patients (60% with ANI) 13. This is likely to be related to differences in inclusion criteria between the Simioni et al. study and the NAMACO study. In the former, patients from French‐speaking Switzerland of any age were recruited if they had cognitive symptoms; in the latter, patients from throughout Switzerland were recruited, regardless of cognitive symptoms, provided they were aged ≥ 45 years. Two other cohort studies have examined the association between answers to the EACS screening questions and NCI. The British Pharmacokinetic and Clinical Observations in People over Fifty (POPPY) study assessed cognitive function in 290 patients, with a median age of 57 years, using a computerized battery (CogState™, London, UK) and found a weak association between NCI, defined according to Frascati and other criteria, and patient‐reported outcome measures including answers to the EACS screening questions 19. The Dutch TREVI study examined cognitive function in 388 patients, with a mean age of 48 years, of whom 69 (17.8%) completed a neuropsychological assessment, and reported a sensitivity and specificity of 82% and 24%, respectively, for the EACS screening questions, which changed to 50 and 73% when used with the International HIV Dementia Scale 20. What our data add to these studies is the high number of patients who underwent formal neuropsychological assessment and the strong association between answering yes, definitely to the EACS screening questions and depression. The relationship between NCI and depression in NAMACO study participants is the subject of another paper (Santos et al., unpublished).

The high NPV (0.9) for excluding NCI with functional impairment and non‐HIV‐associated NCI is of unclear clinical value while the Frascati diagnosis of ANI is under debate, and especially when this NPV was at the expense of a low PPV (0.3). Individuals with ANI are potentially heterogenous, ranging from near‐normal to near‐MND within the ANI spectrum. If more robust criteria for defining NCI could be agreed upon, for example, based on quantitative neurocognitive domain z‐scores, perhaps the NPV of the EACS screening questions in excluding cognitively normal and near‐normal patients can be reviewed. It should also be noted that patients with more severe NCI (MND or HAD) may not have cognitive complaints through anosognosia 21 and so clinical prudence should be employed when applying any subjective screening tool.

This study has limitations. First, the study was limited to a Swiss cohort and our findings may not apply to other populations. Secondly, NAMACO participants had more cognitive complaints compared to eligible but nonrecruited SHCS patients, suggesting a possible selection bias with complaining patients more agreeable to NAMACO recruitment. Against this limitation, this study did not aim to examine the prevalence of NCI or of cognitive complaints but the association between the two. Finally, while the Frascati criteria are currently the only published criteria arrived at by consensus, we acknowledge that using these criteria to classify NCI in our population may have rendered the EACS screening questions less sensitive or specific than they might be were other measures of NCI severity to be applied.

We conclude that the EACS screening questions had an NPV of 0.7 for excluding HIV‐associated NCI in NAMACO study participants using Frascati criteria. It remains to be seen whether the PPV and NPV of these questions improve if patients are classified according to other, yet to be defined, NCI severity criteria. Currently, these questions lack sensitivity and specificity as a tool to guide clinicians as to which patients should be referred for formal neuropsychological testing.

Disclaimer

The opinions expressed in this article are those of the authors and do not necessarily represent those of Viiv. Viiv had no role in the study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Author contributions

MC, RDP and AC designed the study. MM, IL and IN finalized the neuropsychology database. IL performed the statistical analysis. MC and RDP supervised the study. KEAD, IN and MM wrote the manuscript. All investigators contributed to data collection and interpretation, reviewed drafts of posters and the manuscript, and approved the final manuscript.

Fig. S1 ROC curves summing the three European AIDS Clinical Society (EACS) questions (never = 0, hardly ever = 1, and yes, definitely = 2) and taking integer values between 0 and 6 as a pseudo‐continuous means of detecting or excluding neurocognitive impairment (NCI) (ROC curve A), HIV‐associated NCI (ROC curve B), and symptomatic NCI or NCI with confounding factors (other) (ROC curve C).

Click here for additional data file.

Table S1 The seven cognitive domains examined and the neuropsychological tests performed in the standardized neurocognitive assessment of all patients enrolled in the Neurocognitive Assessment in the Metabolic and Aging Cohort (NAMACO) study

Click here for additional data file.

Click here for additional data file.