Emotional context can reduce the negative impact of face masks on inferring emotions.

While face masks prevent the spread of disease, they occlude lower face parts and thus impair facial emotion recognition. Since emotions are often also contextually situated, it remains unknown whether providing a descriptive emotional context alongside the facial emotion may reduce some of the negative impact of facial occlusion on emotional communication. To address this question, here we examined how emotional inferences were affected by facial occlusion and the availability of emotional context. Participants were presented with happy or sad emotional faces who were either fully visible or partially obstructed by an opaque surgical mask. The faces were shown either within an emotionally congruent (e.g., "Her cat was found/lost yesterday afternoon") or neutral ("Get ready to see the next person") context. Participants were asked to infer the emotional states of the protagonists by rating their emotional intensity and valence. Facial occlusion by masks impacted the ratings, such that protagonists were judged to feel less intense and more neutral emotions when they wore masks relative to when their face was fully visible. Importantly, this negative impact of visual occlusion by mask was reduced but not fully eliminated when the faces were presented within a congruent emotional context. Thus, visual occlusion of facial emotions impairs understanding of emotions, with this negative effect of face masks partially mitigated by the availability of a larger emotional context.

Introduction

With the onset of the COVID-19 pandemic, the world saw an unprecedented but necessary widespread adoption of face masks. While masks provide much needed protection again virus spread (Eikenberry et al., 2020; Leung et al., 2020; Prather et al., 2020), they also present challenges to visual social communication as they obstruct approximately 60–70% of the face parts needed for socioemotional messaging (e.g., Carbon, 2020; Mheidly et al., 2020; Molnar-Szakacs et al., 2021 for reviews). Faces are some of most important social stimuli we encounter, and humans readily utilize visible cues from faces to recognize emotions in others (Hugenberg and Wilson, 2013). It has long been demonstrated that facial expressions provide a quick and easy way to extract information about others’ emotional states (e.g., Ekman, 1999), with the ability to read these expressions associated with increased levels of several facets of overall social functioning (e.g., Leppänen and Hietanen, 2001; Addington et al., 2006) including prosocial behavior (Marsh et al., 2007), social approach (Williams et al., 2014), and empathy (Besel and Yuille, 2010).

Many social judgments are made from facial cues (e.g., Klapper et al., 2016). It is thus unsurprising that face masks have been shown to impact many of these judgments, including reducing perceived closeness (Grundmann et al., 2021), increasing perceived attractiveness (Hies and Lewis, 2022; Parada-Fernández et al., 2022) and either increasing (Cartaud et al., 2020; but see Grundmann et al., 2021) or decreasing (Biermann et al., 2021; Gabrieli and Esposito, 2021) perceived trustworthiness. The alterations in such second-order trait perception likely stem from the obstruction of the visual information from the lower face cues needed for basic processes that inform these judgments, such as emotion recognition (Carbon, 2020; Grundmann et al., 2021; Carbon and Serrano, 2021; Williams et al., 2021; Grenville and Dwyer, 2022; Kim et al., 2022; McCrackin et al., 2022a; Parada-Fernández et al., 2022). Indeed, emotion recognition performance for faces obstructed by face masks can decline from 10 to 45% depending on the emotional expression (Carbon, 2020; McCrackin et al., 2022a). That is, recognition of emotional expressions thought to have particularly diagnostic lower face features like disgust and anger (e.g., Ekman, 1999; Smith et al., 2005; Blais et al., 2012; Kret and de Gelder, 2012; Wegrzyn et al., 2017) is the most impacted by lower face occlusion while recognition of expressions with diagnostic upper face regions like fear and surprise is least impacted by lower face occlusion (Carbon, 2020; Carbon and Serrano, 2021; Williams et al., 2021; Grenville and Dwyer, 2022; Kim et al., 2022; McCrackin et al., 2022a).

While the impact of masks on basic emotion recognition is clear (Carbon, 2020; Grundmann et al., 2021; Carbon and Serrano, 2021; Williams et al., 2021; Grenville and Dwyer, 2022; Kim et al., 2022; McCrackin et al., 2022a), recognition of emotional expressions is typically situated within a broader emotional context (Wieser and Brosch, 2012). For example, one might perceive a smile alongside a joke being told or hearing good news being shared. In other words, making emotional inferences requires not only emotional information from faces (Baron-Cohen and Cross, 1992; Clark et al., 2008; Mier et al., 2010; Decety et al., 2015; Stewart et al., 2019) but also contextual information like emotional prosody, body language, prior knowledge, emotional understanding, and/or emotional context (Wieser and Brosch, 2012 for a review).

Here we sought to examine the role of emotional context in making emotional inferences from faces occluded by face masks. Can availability of emotional context ameliorate the negative impact of facial occlusion on emotional communication? To address this question, we asked participants to judge the emotions of protagonists who displayed happy and sad facial expressions and either wore a surgical mask or had their face visually unobstructed. Critically, on half of the trials, the protagonists were presented within a congruent emotional context—a written sentence describing a happy or sad event happening to the protagonist. In the other half of trials, the protagonists were presented within a neutral context—a written sentence informing participants to get ready for the next trial. On each trial, participants rated the intensity and valence of the protagonist’s emotion. Intensity refers to degree of arousal, while valence refers to the degree of pleasantness. The Circumplex theory of emotion (Russell, 1980) suggests that intensity and valence constitute two unique dimensions of affective experience. There is both behavioral and neural evidence to suggest that these dimensions can be dissociated (e.g., Anderson and Sobel, 2003; Kensinger and Corkin, 2004; Colibazzi et al., 2010). For example, strong positive valence can either be paired with high intensity in the experience of happiness or low intensity in the experience of serenity. Thus we decided to examine how facial obstruction by face masks impacted both aspects of affective perception. We also reasoned that using ratings of valence and intensity as in our previous work (see also McCrackin and Itier, 2021; McCrackin et al., 2022a) would avoid potential ceiling effects that may occur from the utilization of forced choice paradigms typically used in emotion recognition given that here we did not only present facial emotional expressions, but also emotional sentences with clear emotional content.

Following from past work (e.g., McCrackin et al., 2022b), we expected to observe diminished (i.e., more neutral) ratings of the protagonists’ emotions when they wore masks. However, if this negative effect of facial obstruction was modulated by the availability of emotional context, we expected to find higher emotional ratings for faces wearing masks in conditions in which congruent context was provided relative to conditions in which no context was provided. Our data supported these predictions.

Methods

This study was pre-registered.[1] Anonymized and summarized data are available at https://osf.io/9bmr3/?view_only=e0871f7add364e378eccd9920f60d98b.

Participants

Seventy undergraduate students participated for course credit and were included in the analysis (66 female, 3 male, 1 other; Mean age: 20.41, SE = 0.13).[2] Sample size was pre-registered and determined with a conservative power analysis based on our previous work with face masks and emotion recognition (McCrackin et al., 2022a) and affective theory of mind (McCrackin et al., 2022b). Participants provided informed consent and the McGill University research ethics board approved the study.

Apparatus and stimuli

The experiment took place online via Testable[3] with stimuli scaled to fit each participant’s personal computer screen. Sample face stimuli are shown in Figure 1A. Images of happy and sad face stimuli were obtained for 20 male and 20 female identities from the FACES (Ebner et al., 2010) and Karolinska Directed Emotional Faces (KDEF; Lundqvist et al., 1998) databases, which have independently validated that these images depict facial expressions with high recognizability (Goeleven et al., 2008; Ebner et al., 2010).[4] For the Mask condition, a photograph of a surgical mask was applied to each face using Adobe Photoshop CS6 and scaled such that the mask spanned the lower edge of the chin, the bridge of the nose, and the edges of the cheeks.

FIGURE 1

(A) Example face stimuli. (B) Example trial progression. In the Context condition, an emotional sentence congruent with the face emotion (either happy or sad) was provided at the start. In the No context condition, the sentence “Get ready for the next person to appear” was presented instead. Following the emotional face image, participants rated the intensity and valence of the protagonist’s emotional state using two separate scales (valence and intensity).

As depicted in Figure 1B, each face stimulus was preceded by a sentence. In the Context condition, the sentence described a face expression within a congruent happy or sad emotional event happening to the protagonists (e.g., “Her pet cat was found/lost yesterday afternoon”). Male and female happy and sad variations of 12 emotional sentence themes from McCrackin and Itier (2021) were presented,[5] as they have repeatedly been demonstrated to elicit the expected emotional responses (McCrackin and Itier, 2021; McCrackin et al., 2022b). In the No Context condition, the sentence “Get ready for the next person to appear” was displayed. This sentence had the same number of syllables as the emotional sentences and similarly referenced the protagonist but did not provide any emotional information.

Design and procedure

The study was a repeated measures design with three factors---Context (2; Context; No Context), Emotion (2; Happy, Sad), and Mask (2; Mask, No Mask). Context manipulated whether an emotional sentence (Context) or neutral sentence (No Context) preceded the presentation of a protagonist’s emotional face. This variable was blocked, such that half of the testing blocks (i.e., 4) provided emotional context (Context blocks) and half did not (No Context blocks), with the block order and trials within the blocks randomized. The factor of Emotion manipulated whether the face depicted a Happy or Sad facial expression. Facial expression and emotional context were congruent during the Context condition such that happy context sentences were always paired with happy expressions and sad context sentences were paired with sad expressions.[6] The Mask factor manipulated whether the face wore a face mask (Mask) or not (No Mask).

Manipulating these three factors yielded 8 experimental conditions. Each condition was sampled 24 times for a total of 192 trials divided across 8 testing blocks. The same face identities were presented in the Context and No Context blocks so that the impact of emotional context could be examined without changing any other variables (i.e., participants saw the same face image once with context and once without context). All conditions were equiprobable and presented using a pseudorandom sequence.

Figure 1B illustrates a typical trial. Participants were first shown either an emotional context sentence (Context condition) or a neutral sentence (No Context condition) for 4,000 ms. A 200 ms fixation cross preceded a presentation of the Happy or Sad emotional face either wearing a mask (Mask) or not wearing a mask (No Mask) for 2,000 ms. After the image presentation, participants were asked to use a 9-point Likert scale to rate the protagonists’ (i) emotional intensity ranging from 0/very un-intense to 9/very intense, and (ii) emotional valence from 0/very negative to 9/very positive on separate screens and were given unlimited time to make each response. These two rating scales were designed to probe the affective dimensions of valence (pleasure) and arousal (intensity) theorized by the Circumplex model of emotion (Russell, 1980) to represent dissociated components of the emotional experience.

Results

Mean ratings of the protagonists’ emotional intensity and valence were calculated for each participant. Then, two separate repeated measures ANOVAs were run on each dependant variable (i.e., intensity and valence) with Context (Context, No Context), Emotion (Happy, Sad), and Mask (Mask, No mask) included as factors. Follow-up two-tailed paired-test tests were performed where required, with Bonferroni correction applied to the nominal α = 0.05 level.

To remind, we hypothesized that if context contributes significantly to emotional understanding, availability of a congruent emotional context should provide mitigating effect under conditions in which visual facial cues are unavailable due to facial obstruction by mask.

Intensity

Confirming the efficacy of our context manipulation, a main effect of Context indicated that intensity ratings were overall higher when a protagonist was presented within an emotional context relative to no context [F(1, 69) = 48.92, MSE = 0.60, p = 1.36 × 10–9, η = 0.42]. As depicted in Figure 2A, a Context × Emotion interaction indicated that availability of an emotional context increased intensity ratings for happy emotions more than for sad ones [F(1, 69) = 7.87, MSE = 0.18, p < 0.007, η2 = 0.10], although the effect was significant for both happy [t(69) = 8.13, p = 1.14 × 10–11, SE = 0.080 d = 0.97] and sad trials [t(69) = 4.46, p = 3.1 × 10–5, SE = 0.069, d = 0.53].

FIGURE 2

Intensity ratings. The mean and median for each condition are indicated with a black dot and solid line, respectively, and each participants’ data point is plotted. Impact of (A) emotional context and (B) mask on intensity ratings for each emotion. (C) The impact of masks on intensity ratings as a function of Emotion and Context. (D) The overall mask impact (unmasked intensity—masked intensity ratings) with and without context. Note that the rating of 1 represents the lowest emotional intensity.

Replicating previous work indicating that facial obstruction by masks alters emotional inferences (McCrackin et al., 2022b), a main effect of Mask indicated that faces wearing masks were judged to feel less intense emotion than those not wearing masks [F(1, 69) = 84.87, MSE = 0.73, p = 1.23 × 10–13, η2 = 0.55]. As shown in Figure 2B, a Mask x Emotion interaction indicated that this reduction in perceived emotional intensity for faces wearing masks was also larger for happy than for sad emotions [F(1, 69) = 38.30, MSE = 0.15, p = 3.80 × 10–8, η2 = 0.36], although wearing masks reduced emotional intensity ratings for both happy [t(69) = −10.35, p = 1.09 × 10–15, SE = 0.083, d = −1.24] and sad emotions [t(69) = −6.18, p = 3.92 × 10–8, SE = 0.074, d = −0.74].

Finally, we predicted that adding emotional context may be able to reduce some of the negative impact of facial occlusion by masks on emotional inferences. Indeed, and as depicted in Figure 2C, there was a significant Context × Mask interaction [F(1, 69) = 16.39, MSE = 0.065, p = 1.33 × 10–4, η2 = 0.19]. While there was an effect of mask for both Context [t(69) = −7.93, p = 2.68 × 10–11, SE = 0.072, d = −0.95] and No Context conditions [t(69) = 9.67, p = 1.81 × 10–14, SE = 0.072, d = −1.16], the effect of visual occlusion by Masks (computed as Intensity rating without masks—Intensity rating with masks) was reduced by approximately 23% when Context was provided [t(69) = −4.05, p = 1.33 × 10–4, SE = 0.043, d = −0.48], as depicted in Figure 2D. The three-way interaction between Context, Emotion, and Mask was not significant [F(1, 69) = 0.005, MSE = 0.08, p = 0.94, η < 0.001].

Valence

Similarly to intensity, and confirming the efficacy of the emotional manipulation, a main effect of Emotion valence indicated that protagonists were judged to feel more positive in the happy emotional condition compared to the sad emotional condition [F(1, 69) = 1537.32, MSE = 1.97, p = 6.75 × 10–49, η2 = 0.96]. A main effect of Mask reflected that protagonists were also judged to feel more positive when their face was visually unobstructed as opposed to when they were wearing masks [F(1, 69) = 33.29, MSE = 0.11, p = 2.06 × 10–7, η2 = 0.33], although this was qualified by a Mask by Emotion interaction discussed below.

As shown in Figure 3A, significant Context x Emotion interaction indicated that availability of emotional context led to heightened assumptions of the individual feeling the inferred emotion [F(1, 69) = 94.41, MSE = 0.26, p = 1.51 × 10–14, η2 = 0.58]. That is, availability of an emotional context increased valence ratings for happy trials (p < 0.001) and decreased valence ratings for sad trials (p < 0.001), with the magnitude of this effect no different between the happy and sad emotion conditions (p = 0.86).

FIGURE 3

Valence ratings. The mean and median for each condition are indicated with a black dot and solid line, respectively, and each participants’ data point is plotted. Impact of (A) emotional context and (B) mask on valence ratings for each emotion. (C) The impact of masks on valence ratings as a function of emotion and context. (D) The overall impact of masks (unmasked valence—masked valence ratings for happy emotion, and masked valence—unmasked valence for sad emotion) for positive happy and sad negative emotions as a function of context. The neutral valence point (rating 5) is indicated by the dotted line.

Further, as depicted in Figure 3B, a Mask x Emotion interaction [F(1, 69) = 41.06, MSE = 0.22, p = 2.50 × 10–21, η2 = 0.73] indicated that individuals wearing masks were rated as feeling more neutral emotions than those not wearing masks, replicating our previous work (McCrackin et al., 2022b). For happy trials, protagonists wearing masks were rated with lower valence ratings [t(69) = −12.77, p = 7.58 × 10–20, SE = 0.055, d = −0.1.53], while for sad trials, they were rated with higher valence ratings [t(69) = 9.18, p = 1.43 × 10–13, SE = 0.042, d = 1.10]. The impact of masks on valence ratings was larger for happy trials than for sad trials [t(69) = 5.77, p = 2.06 × 10–7, SE = 0.057, d = 0.69].

Finally, as illustrated in Figure 3C, there was also a three-way interaction between Context, Emotion, and Mask [F(1, 69) = 48.74, MSE = 0.04, p = 1.44 × 10–9, η2 = 0.41]. As shown, this interaction was driven by the availability of emotional context reducing the impact of masks on valence ratings during both sad [t(69) = −9.26, p = 1.27 × 10–13, SE = 0.045, d = −1.11] and happy [t(69) = 8.76, p = 8.30 × 10–13, SE = 0.048, d = 1.05] emotions, with a similar magnitude [t(69) = 0.17, p = 0.86, SE = 0.036, d = 0.02]. Proportionately, availability of context reduced the negative effect of facial occlusion on valence ratings, plotted in Figure 3D, by 51% for sad emotions and by 24% for happy emotions.

To summarize, providing emotional context was overall associated with emotional judgments of protagonists feeling more intense emotions and stronger emotional valence. When protagonists wore face masks they were judged as feeling less intense and weaker emotional valence, particularly for happy emotions. The availability of emotional context significantly reduced the negative impact of face masks on both ratings of intensity and valence, with the magnitude of this reduction ranging from 23% up to 51%.

Discussion

Lower face occlusion with face masks has been shown to impair our ability to recognize facial emotional expressions (Carbon, 2020; Grundmann et al., 2021; Carbon and Serrano, 2021; Williams et al., 2021; Grenville and Dwyer, 2022; Kim et al., 2022; McCrackin et al., 2022a), prompting concerns about the effectiveness of social interactions in masked situations (Mheidly et al., 2020; Molnar-Szakacs et al., 2021). However, during real life social interactions, emotional expressions are typically experienced within a broader emotional context that might compensate for the lack of lower face cues (Wieser and Brosch, 2012). Thus, it is important to consider how emotional context may affect emotional processing under conditions when visual emotional information from faces may not be available.

To investigate this question, we presented participants with images of emotional faces who either wore masks or had their faces visually unobstructed. Critically, the protagonists were presented within either an emotional or neutral context. Participants were asked to rate the emotional state of each protagonist. The data indicated overall reduced emotional processing from faces wearing masks. Availability of emotional context reduced, but did not fully reverse, this negative impact of facial occlusion. Next, we discuss two points relating to these data.

First, replicating and extending existing reports (Carbon, 2020 for a review) we found an impact of face occlusion by masks on emotional inferences, both when emotional context was available and when it was not available. When protagonists wore masks they were judged as feeling more neutral and less intense emotion. This finding dovetails with recent work from both emotion recognition (Carbon, 2020; Grundmann et al., 2021; Carbon and Serrano, 2021; Williams et al., 2021; Grenville and Dwyer, 2022; Kim et al., 2022; McCrackin et al., 2022a) and emotional valence and intensity paradigms (McCrackin et al., 2022b) to suggest that face occlusion by masks significantly impacts not only basic emotion recognition but also judgments of emotional states that integrate both emotional expressions and contextual information.

Of note here is our finding that face covering by masks seemed to impact happy emotional inferences more than sad ones. We recently found a similar asymmetry for understanding happy emotions to be more impacted than understanding sad emotions when we asked participants to both infer and share emotions with protagonists wearing face masks (McCrackin et al., 2022b). One explanation for this finding is that face masks impact the perception of happy expressions more than they impair the perception of sad ones. Consequently, this basic emotion perception impairment may exert a larger downstream effect on happy emotional inferences. While some recent studies have reported greater impact in recognizing sad relative to happy expressions from masked faces (Carbon, 2020; Williams et al., 2021; Kim et al., 2022; McCrackin et al., 2022a), it is important to highlight that these studies have mainly examined overall emotion recognition accuracy (i.e., percent correct identifications) and happy facial expressions are well known to be the easiest emotion to recognize (i.e., the so-called happy superiority effect in emotion recognition; e.g., Neath, 2012; for a review; Svard et al., 2012; Švegar et al., 2013). It is possible that happy expressions are still easily recognized from faces wearing masks due to ceiling effects, but the perceived intensity and valence of those happy expressions remain more strongly impacted than perceived intensity and valence of sad expressions. In line with this point, our data also suggest that perception of happy facial expressions was impacted more by lower face occlusion than the perception of sad facial expressions. This is likely because the diagnostic smile is fully covered by the face mask, while the eyes remain unobstructed as a clear diagnostic feature for sadness (e.g., Ekman, 1999; Smith et al., 2005; Blais et al., 2012; Kret and de Gelder, 2012; Wegrzyn et al., 2017). Future studies are needed to further understand the links between different facets of emotional inferences under conditions in which facial cues may not be readily available.

Second, we also found that availability of an emotional context reduced the negative impact of facial occlusion by masks on ratings of both happy and sad emotional states. That is, masked individuals received closer ratings to the unmasked individuals when their images were paired with a congruent emotional context as opposed to when their images were paired with a neutral context. Thus, while contextual information was not necessary for understanding the general emotional state, it modulated the extent to which the inferred emotional state was impacted by visual occlusion of face parts. The reduction of the mask impact ranged from 23% for intensity ratings, and 51% for sad valence ratings, to 24% for happy valence ratings. As such, this suggests that providing contextual statements during social interactions while protagonists wear masks may provide a relatively simple way in which the impact of facial occlusion by masks can be reduced.

There are a few points that warrant further investigation. First, our study used static images with photoshopped masks. Since dynamicity typically facilitates emotion recognition (e.g., Weyers et al., 2006; Enticott et al., 2014; but see Gold et al., 2013), dynamic emotional stimuli may facilitate or change emotion recognition while protagonists wear masks. It is also possible that actors wearing masks may change how they emote with their upper face features, as suggested by Okazaki et al. (2021). Future work is needed to understand the commonalities and differences in emotional communication in static and dynamic experimental conditions. Second, here we focused on understanding how facial occlusion and emotional context impacted understanding of happy and sad emotional expressions, but previous work has shown that face masks impair the perception of all six basic emotions (e.g., McCrackin et al., 2022a). An interesting next step would be to investigate whether availability of a congruent context can also reduce the impact of masks on recognition of other basic emotions as well. It is possible that understanding emotions with diagnostic upper face features (e.g., fear) may not be as impacted by contextual manipulations as understanding emotions with diagnostic lower face features (e.g., disgust) when lower visual features are occluded by masks. Opposite results may be expected for eye coverings. Third, we focused on the use of face masks as visual occluders, but there is evidence to suggest that the type of face occlusion may matter (Wang et al., 2015). For example, Fischer et al. (2012) reported that covering the lower face with a niqab led to a bias to perceive emotions as more negative, while Kret and Fischer (2018) reported key differences in how emotions were recognized when lower faces were covered by a western winter scarf relative to a niqab. Face masks themselves may now have implied positive or negative responses depending on the participant and their experiences, so future studies can examine the role of the type of face covering in social attribution effects. Finally, individual participant factors may have also played a role in our results. For example, our sample was mostly western, female skewed, and educated young adults. Individual factors such as gender (e.g., Hoffmann et al., 2010; Abbruzzese et al., 2019; Gamsakhurdashvili et al., 2021), age (Abbruzzese et al., 2019; Carbon, 2020), and cultural experience (Elfenbein et al., 2002) have been shown to play a role in emotion recognition. For example, women appear to be better at detecting subtle facial emotions (Hoffmann et al., 2010), and thus our participants may have better emotions recognition overall (from masked and unmasked faces). An important next step would be to examine if these results generalize to a more diverse sample.

In summary, wearing face masks lowers our ability to infer emotional states in others, with inferences about happy emotional states affected more than inferences about sad emotional states. This negative impact of visual occlusion by face masks can be reduced by incorporating verbal statements which provide congruent emotional context.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found below: https://osf.io/9bmr3/?view_only=e0871f7add364e378eccd9920f60d98b.

Ethics statement

The studies involving human participants were reviewed and approved by the McGill University REB. The patients/participants provided their written informed consent to participate in this study. Written informed consent was obtained from the individual(s) for the publication of any identifiable images or data included in this article.

Author contributions

SM and JR were involved in the early conceptualization and experimental design. SM oversaw the study programming and carried out data processing and analysis with advice from JR. SM wrote the initial manuscript draft, which was later revised and approved by both authors.