New insights into facial emotion recognition in Parkinson's disease with and without mild cognitive impairment from visual scanning patterns.

BACKGROUND: Recognizing emotional facial expressions is crucial for social interactions. Cognitive impairment and oculomotor abnormalities are common features of Parkinson's disease (PD) which may contribute to the performance in facial emotion recognition (FER) in PD.
OBJECTIVE: The aim of this study was to analyze eye movement behavior during a facial emotion recognition (FER) task with respect to cognitive state in PD patients and healthy controls.
METHODS: Eye movements of 24 non-demented, non-depressed PD patients (12 with intact cognitive functions and 12 with Mild Cognitive Impairment (MCI) according to MDS task force criteria level 2), and 12 age-, sex and education-matched healthy controls were recorded during visual exploration of 28 emotional (happiness, surprise, disgust, anger, fear and sadness) and neutral faces. Participants were asked to identify the displayed emotion out of a sevenfold multiple choice question.
RESULTS: PD-MCI patients showed reduced FER with specific impairment of anger recognition. Although the scanned area of PD patients with intact cognition was significantly restricted, they did not differ in FER from healthy subjects. While healthy subjects and cognitively intact PD patients scanned faces with preference for mouth and eyes, patients with PD-MCI tended to look at the center of the face and spent significantly less time fixating the mouth.
CONCLUSIONS: Ineffective visual exploration may contribute to impaired emotion recognition in PD. Visual scanning of emotional faces is altered in PD even in the absence of cognitive impairment. The progression to PD-MCI may result in further deterioration of scanning behavior and FER impairment.

Introduction

Facial emotion recognition (FER) is an essential skill in social interactions among human beings. [1] Studies assessing FER in Parkinson's disease (PD) reported contradictory results which may be due to several reasons. First, earlier studies did not consider cognitive impairment, disease severity, medication state or psychiatric co-morbidity, especially depression. Furthermore, the sensitivity of study methods varied with difficulty of task design (e.g. identification of the correct emotion vs. discrimination between two emotions). A recent review concluded in favor of a specific impairment of FER in for negative basic emotions. [2] However, the underlying mechanism of this bias in emotion recognition is still not well understood.

FER is dependent on the visual perception and shifting attention to a combination of typical clues in a face expressing a specific emotion. [3] Although deficits in executive functions and attention are frequent in PD [4], the impact of cognitive impairment on FER and the distinct roles of different cognitive domains remain unclear so far. A study with a rather large sample size of 97 PD patients demonstrated that patients with Mild Cognitive Impairment (MCI) show more severe FER deficits than those with intact cognitive ability. [5] The possible role of visuo-spatial deficits in FER of PD patients is supported by a study which showed that FER of anger correlated with the ability to distinguish perfect from imperfect circles. [6] However, impairment of FER in PD may be present even after controlling for cognitive dysfunction. [7]

Besides the potential influence of higher cognitive control deficits on FER, impaired basic oculomotor performance in PD, which was demonstrated in a large number of studies on reflexive and voluntary saccades, may also play a role. [[8], [9], [10], [11]] In a visual exploration study, subjects with PD scanned smaller areas of images than healthy individuals while executing fewer and smaller saccades, as well as longer fixations. [12] A second study on visual scanning in PD highlighted the impact of cognitive impairment on effectiveness of visual search strategies. [13] Despite the possible importance of visual scanning deficits to social and emotional functioning in PD, only one study investigated eye movements during FER in PD so far: In a cohort of 16 patients, Clark et al. found that the fixation duration during FER correlated with executive functions. [13]

The objective of the current study is to explore the relationship between FER and eye movements during visual scanning of emotional faces in PD with respect to cognitive functions.

Material and methods

The study was approved by the local ethical committee at the University Hospital Marburg and was conducted in accordance with the Declaration of Helsinki. All participants gave written informed consent.

24 patients diagnosed with PD according to UK Brain Bank Criteria and 12 healthy control subjects (HC) with normal or corrected-to-normal visual acuity were recruited for this study. Persons with other relevant ophthalmological, neurological, neurodegenerative or psychiatric disorders, in particular depression (BDI score > 13), were excluded. 12 HC, 12 PD patients with intact cognitive ability (cognitively normal, PD-CN) in a comprehensive neuropsychological test battery and 12 patients with PD-MCI were cross-matched for age (+/− five years), years of education (+/− one year), sex and disease duration (+/− one year).

All PD patients performed the FER task after intake of their regular dose of dopaminergic medication. Motor performance was assessed using the motor section of the Movement Disorders Society - Unified Parkinson's disease Rating Scale (UPDRS). [14] Each subject underwent a comprehensive neuropsychological testing. According to the recommendations of the MDS Task Force Level II Criteria for the diagnosis of Mild Cognitive Impairment, [15,16] the test battery included two tests for each of the five cognitive domains attention (Trail Making Test A, digit span), executive functions (Trail Making Test B, Frontal Assessment Battery), language (semantic verbal fluency in the supermarket task, phonemic verbal fluency for “F” words), memory (immediate and delayed recall of the 10-items list from the DemTect and recall of Rey Complex Figure) and visual-spatial function (clock drawing and copying of Rey Complex Figure) (Table 1). A performance below two standard deviations (SD) compared to age-matched reference groups in the literature was defined as pathological [[17], [18], [19], [20], [21]].

Table 1

Clinical characteristics and results.

	HC (n = 12)			PD-CN (n = 12)			PD-MCI (n = 12)			HC vs. PD-CN vs. PD-MCI
	mean	SD	n < 2 SD	mean	SD	n < 2 SD	mean	SD	n < 2SD	p	Multiple comparison
Age (years)	61.8	11.1	–	63.4	9.5	–	63.8	10.2	–	0.9a
Years of education	13.3	1.8	–	13.9	1.8	–	13.1	1.3	–
Disease duration (years)	–	–	–	8.8	4.4	–	7.8	3.5	–	0.6t
LEDD (mg per day)	–	–	–	717	236	–	730	401	–	0.9t
MDS-UPDRS III	–	–	–	25.5	10.7	–	33.6	13.5	–	0.1t
MoCA (max. 30)	26.7	2.4	–	27.1	2.1	–	22.3	1.2	–	0.0002k	HC vs. PD-CN p > 0.99HC vs. PD-MCI p = 0.0002PD-CN vs. PD-MCI p = 0.0007
Trail Making Test A (s)	30.8	9.5	0	33.3	9.9	1	64.2	39.3	5	0.004k	HC vs. PD-CN p > 0.99HC vs. PD-MCI p = 0.007PD-CN vs. PD-MCI p = 0.02
Digit span (n)	4.9	0.9	0	4.8	1.1	1	4.5	1.0	1	0.6k
Trail Making Test B (s)	75.8	33.8	0	67.3	21.7	0	183.6	78.8	10	<0.0001a	HC vs. PD-CN p = 0.86HC vs. PD-MCI p = 0.002PD-CN vs. PD-MCI p = 0.001
Frontal Assessment Battery (max. 18)	16.8	1.7	1	17.6	0.6	0	14.0	3.1	6	0.008k	HC vs. PD-CN p > 0.99HC vs. PD-MCI p = 0.07PD-CN vs. PD-MCI p = 0.008
Phonematic verbal fluency (n; “F”)	12.9	3.6	0	14.3	3.7	1	10.5	2.5	1	0.04k	HC vs. PD-CN p = 0.8HC vs. PD-MCI p = 0.2PD-CN vs. PD-MCI p = 0.03
Semantic verbal fluency (n; “supermarket”)	26.2	4.6	0	24.5	4.6	0	20.3	5.3	1	0.03k	HC vs. PD-CN p > 0.99HC vs. PD-MCI p = 0.03PD-CN vs. PD-MCI p = 0.3
10-item word list (immediate and delayed recall) (n, max. 20)	13.2	4.5	0	13.9	4.1	0	9.4	4.2	4	0.04a	HC vs. PD-CN p = 0.96HC vs. PD-MCI p = 0.2PD-CN vs. PD-MCI p = 0.05
Rey Complex Figure copying (score, max. 36)	34.3	2.7	0	31.8	7.2	0	26.1	10.2	0	0.2k
Rey Complex Figure delayed recall (score, max. 36)	19.7	6.5	0	15.6	6.5	0	10.2	7.8	3	0.01a	HC vs. PD-CN p = 0.4HC vs. PD-MCI p = 0.01PD-CN vs. PD-MCI p = 0.2
Clock drawing (Shulman score, max. 6)	1	0	0	1.5	1.0	1	2.3	1.1	8	0.005k	HC vs. PD-CN p = 0.6HC vs. PD-MCI p = 0.003PD-CN vs. PD-MCI p = 0.1
Total FER error rate	0.21	0.08	–	0.26	0.11	–	0.32	0.10	–	0.049a	HC vs. PD-CN p = 0.6HC vs. PD-MCI p = 0.04PD-CN vs. PD-MCI p = 0.6
Neutral FER error rate	0.06	0.11	–	0.06	0.11	–	0.22	0.30	–	0.3k
Happiness FER error rate	0	0	–	0.04	0.09	–	0.09	0.13	–	0.2k
Surprise FER error rate	0.13	0.16	–	0.08	0.12	–	0.13	0.15	–	0.8k
Fear FER error rate	0.44	0.32	–	0.48	0.31	–	0.49	0.30	–	0.9k
Sadness FER error rate	0.38	0.22	–	0.60	0.22	–	0.49	0.31	–	0.1a
Anger FER error rate	0.21	0.22	–	0.33	0.24	–	0.48	0.21	–	0.038k	HC vs. PD-CN p = 0.5HC vs. PD-MCI p = 0.03PD-CN vs. PD-MCI p = 0.7
Disgust FER error rate	0.27	0.19	–	0.23	0.26	–	0.31	0.35	–	0.8k
Number of fixations per face	7.4	2.6	–	5.7	3.1	–	5.4	2.3	–	0.2a
Fixation duration per face (ms)	2144	479	–	2015	619	–	2020	854	–	0.9a
Mean fixation duration (ms)	290.0	73.2	–	345.9	124.8	–	398.1	128.1	–	0.044k	HC vs. PD-CN p = 0.5HC vs. PD-MCI p = 0.04PD-CN vs. PD-MCI p = 0.9
Mean saccade amplitude (°)	3.17	0.40	–	2.84	0.41	–	2.83	0.54	–	0.5a
Relative scanned area	0.49	0.18	–	0.30	0.10	–	0.42	0.17	–	0.029a	HC vs. PD-CN p = 0.03HC vs. PD-MCI p = 0.8PD-CN vs. PD-MCI p = 0.2
Relative fixation duration on upper face ROI	0.53	0.20	–	0.53	0.19	–	0.68	0.18	–	0.1a
Relative fixation duration on lower face ROI	0.42	0.22	–	0.42	0.17	–	0.20	0.17	–	0.0053k	HC vs. PD-CN p > 0.99HC vs. PD-MCI p = 0.02PD-CN vs. PD-MCI p = 0.01

t = Student's t-test, a = Welch ANOVA test followed by Holm-Sidak's test k = Kruskal-Wallis test followed by Dunn's multiple comparison test

The eye tracking session took place in a dimly lit room and stimuli were presented on a computer monitor in a viewing distance of 60 cm. Participants were sittingseated in a comfortable chair with their head stabilized by a chin rest and a forehead bar. A video-based eye tracker (EyeLink 1000 plus, SR Research Ltd.) recorded the monocular position of the right eye with a sampling rate of 500 Hz. Stimuli were presented using the open-source program Open Sesame. [22] Each stimulus consisted of a fixation cross for drift correction followed by one of a total of 28 static photographs of emotional faces (half male, half female) from the validated Radboud database. [23] Four pictures of each of the six basic emotions anger, fear, happiness, surprise, sadness and disgust and four neutral faces were presented for 5000 ms. The size of the image was chosen to resemble a real face in the same viewing distance. After each picture, the participant was asked to choose one emotion out of a sevenfold multiple choice question.

A parsing system is incorporated in the EyeLink 1000 plus that intersects eye position data into saccades, fixations and blinks. This raw data set was analyzed in the statistical computing R using the eyelinker package. [24,25] A 40°/s peak velocity threshold for saccade detection and 60 ms for minimum fixation duration were used. Assessed eye tracking features were mean saccade amplitude, mean and total fixation duration and total number of fixations in the face region of interest (ROI). Furthermore, two ROIs within the face were defined to assess oculomotor outcomes in the upper (eyes) and lower (mouth) region of the faces separately. The ROIs were limited by the lateral edges of the face, the middle of the nasal ridge and the middle of the forehead, respectively the chin (Fig. 1). The scanned area was calculated using a convex hull algorithm that calculated the smallest possible polygon covering all fixations within the face ROI [26]. The area is presented relative to the total area of the face ROI.

Fig. 1

Definition of ROIs for total, lower and upper face regions.

To further illustrate the oculomotor results, heatmaps were created using the ggplot2 package in R [24] showing the distribution of the first five fixations that were performed to scan the four pictures of each emotion (Fig. 2). Since the median numbers of fixations executed within the face ROI in the PD-CN and PD-MCI group were five, only the first five fixations were used for the heatmaps for all three groups.

Fig. 2

FER error rates and heatmaps of frequency of fixations in total and by emotion (projected onto an exemplary picture for illustration), *p < 0.05.

Statistics were executed in Prism 8 (GraphPad Software, LLC). Normal distribution of data was tested with Shapiro–Wilk test. If data did not pass normality test, non-parametric tests were applied. One-way ANOVA, respectively Kruskal-Wallis test, followed by multiple comparisons with Dunn's correction were performed to compare FER as well as oculomotor outcomes between the three groups. False Discovery Rate method was used to correct for multiple testing. [27] All reported p-values are two-tailed and a p-value <0.05 was considered significant.

Results

The variation of FER error rates between different emotions was comparable between the groups, varying between 0% for happiness and up to 44% for fear in HC and between 7% for happiness and 49% for fear and sadness in PD-MCI patients (Fig. 2).

ANOVA revealed a significant group effect for the total FER error rate (F = 3.49, p = 0.049) and multiple comparison resulted in a higher FER error rate in PD-MCI patients than in healthy controls (p = 0.04). By emotion, PD-MCI patients performed significantly lower than HC and PD-CN patients exclusively in the recognition of anger (F = 6.57, p = 0.04, HC vs. PD-MCI p = 0.03).

The analysis of eye movements during visual scanning resulted in a significant group effect for mean fixation duration (F = 6.21, p = 0.04), scanned area (F = 5.05, p = 0.02) and relative fixation time on the lower face ROI (F = 10.50, p = 0.005) (Fig. 3). Mean fixation duration was increased in the PD-MCI group (HC vs, PD-MCI p = 0.04) compared to HC. The relative fixation time in the lower face ROI was significantly shorter in the PD-MCI group than in both HC and PD-CN (HC vs. MCI p = 0.03, PD vs. MCI p = 0.01). The same shift from the mouth to the middle and upper part of the face, in particular to the nose and nasolabial fold, in PD-MCI is also apparent in the distribution of the first five fixations on the face ROI per group shown in Fig. 4. The scanned area was reduced in PD-CN (HC vs. PD-CN p = 0.03), but not in PD-MCI.

Fig. 3

Oculomotor results comparing healthy controls (HC), PD patients with intact cognition (PD-CN) and PD patients with Mild Cognitive Impairment (PD-MCI), *p < 0.05.

Fig. 4

Distribution of the first five fixations on all faces and emotions (projected onto an exemplary picture). The size of the dots represents the fixation duration.

To check for emotion-specific alternations of scanning behavior, we performed a post-hoc comparison of total fixation to the face ROI and scanned areas between the different emotions in the PD group. Here, the sole difference was that PD patients spend significantly less time fixating within the face ROI of sad and angry faces than happy faces (F = 14.79, p = 0.01; sad vs. happy p = 0.04, angry vs. happy p = 0.01).

Discussion

In this study we demonstrated that FER is impaired in PD patients with MCI, in particular for the recognition of anger, and that those patients show an altered visual scanning pattern during FER with reduced focus on the mouth. Although PD patients with intact cognitive ability did not differ from healthy controls in their FER performance, their scanning areas during visual exploration of faces were restricted.

Studies in healthy individuals showed that distinct visual scanning patterns emerge for each emotion, such as a focus on the mouth of happy faces and the eyes of sad and angry faces [28] which is present even when the subject searches for evidence of a specific emotion in neutral faces [29]. This goal-driven strategy in visual scanning for emotion recognition seems to deteriorate with cognitive decline in PD-MCI, since PD-MCI patients show a shift to the middle part of the face as well as an overall deterioration of focus on the mouth. For non-emotional face recognition, such a holistic scanning strategy with focus on midline structures was found to be associated with cognitive decline in healthy older subjects in a recent study. [30] Likewise, the visual exploration strategy seems to be ineffective in PD-MCI since FER decreased, although the overall scanned area did not differ from controls.

We hypothesize that visuo-spatial or executive deficits may lead to impaired selecting and planning of appropriate targets for fixations and, thereby, to a wider scanned area (compared to PD-CN) with less focus on the important features of the face (see Fig. 2, Fig. 4). Furthermore, suboptimal landing of saccades on the next facial feature may increase the processing load that is necessary to integrate the information gathered during this suboptimal fixation, resulting in the prolonged fixation duration in PD-MCI.

In line with our results, earlier studies on volitional eye movements during visual scanning in PD found an increase in mean fixation duration, which was more pronounced in PD patients with MCI. [12,13] Thus, increased fixation durations do not seem to be specific to visual scanning of faces, but may reflect the increased cognitive processing load needed to integrate visual cues and to plan the next saccade.

Although the scanned area of faces was decreased in PD patients with intact cognitive ability, their FER did not differ from healthy control subjects. Studies in healthy individuals showed that the visual scanning pattern has more predictive value in FER of subtle than extreme emotions [31]. While the restricted scanning area of PD patients may still be sufficient for the recognition of the very obvious emotions used in our study, it may become disadvantageous when the emotion intensity is lower. Thus, it would be interesting to investigate the visual exploration of varying emotion intensities in PD patients in the future.

Like the majority of earlier FER studies in PD, we found a specific impairment in the recognition of negative emotions in PD-MCI, here of anger. One may argue that the correct identification of negative emotions is more difficult due to the higher diversity on the spectrum of negative emotions compared to happiness being the only un-ambiguous positive emotion. This explanation is supported by the fact that the FER error rate varied between emotions in all groups and was lowest for happy faces in healthy controls as well. However, we found significant alterations in visual scanning of negative emotions in PD patients: They spend less time scanning sad and angry faces compared to happy faces. Thus, reduced visual exploration of negative emotions, which could be interpreted as an avoidance mechanism, may contribute to impaired recognition of these emotions. Furthermore, impaired visual exploration may have a higher impact on the differentiation of the larger range of negative emotions.

Degeneration of the substantia nigra and the amygdala, which is primarily involved in processing of anxiety and aggression, and thereby of angry and fearful facial expressions, occur simultaneously in Braak's pathological model of PD. Furthermore, specific amygdalar dysfunction has been demonstrated in several imaging studies in PD. [32] An MRI study showed that the identification of anger correlated with the gray matter volume in in the right occipital fusiform gyrus, ventral striatum and subgenual cortex in PD patients [33]. Additionally, FER performance was related to FDG hypometabolism in right and left superior frontal gyrus and bilateral posterior cingulate in a PET-study [34]. Taken together all these results suggest that both PD pathology as well as functional dopamine depletion in basal ganglia, limbic and fronto-striatal pathways may cause a specific impairment in the recognition of negative emotions in PD which may be mediated by reduced visual scanning.

A few limitations of this study need to be discussed: First, the sample size of the study as well as the number of trials per emotion are relatively small which limits the external validity. Patients were tested in on medication state and an effect of levodopa or dopamine agonists on FER may have had an impact on the results. Although we did control for depression, a higher prevalence of apathy in PD may have had an impact on FER as well. Despite matching for disease duration and comparable levodopa equivalent daily dosage (LEDD), patients with PD-MCI had slightly higher mean UPDRS in on medication state. The higher motor burden may reflect a more accelerated disease course and spreading of the underlying pathology in the brain compared to the PD group with intact cognition which may have influenced the results.

In conclusion, this study showed that oculomotor alterations in the visual exploration of emotional faces are evident in PD even before the recognition of facial emotions decreases with progression of cognitive impairment. The findings of specifically impaired anger recognition combined with reduced visual exploration of angry faces support the hypothesis that the impaired FER of negative emotions may have a specific neural correlate in PD.

Since FER plays a crucial role in social interactions, the findings of the current study have clinical implications: Awareness of the specific impairments of FER in PD may help to improve the relationships with care partners and health care professionals. Furthermore, a better understanding of the underlying mechanisms of impaired FER in PD may facilitate the development of neurocognitive training strategies to potentially compensate FER deficits in PD-MCI.

Declaration of competing interest

None.