Compound facial expressions of emotion: from basic research to clinical applications.

Emotions are sometimes revealed through facial expressions. When these natural facial articulations involve the contraction of the same muscle groups in people of distinct cultural upbringings, this is taken as evidence of a biological origin of these emotions. While past research had identified facial expressions associated with a single internally felt category (eg, the facial expression of happiness when we feel joyful), we have recently studied facial expressions observed when people experience compound emotions (eg, the facial expression of happy surprise when we feel joyful in a surprised way, as, for example, at a surprise birthday party). Our research has identified 17 compound expressions consistently produced across cultures, suggesting that the number of facial expressions of emotion of biological origin is much larger than previously believed. The present paper provides an overview of these findings and shows evidence supporting the view that spontaneous expressions are produced using the same facial articulations previously identified in laboratory experiments. We also discuss the implications of our results in the study of psychopathologies, and consider several open research questions.

Introduction

Humans are especially good at expressing emotions through facial expressions.[1-4] The hypothesis is that primates have evolved a sophisticated set of facial muscles that now allows them to convey emotions to observers.[5-8] To date, most research has focused on the study of six emotions typically seen in most cultures: happiness, surprise, sadness, anger, fear, and disgust.[9] Previous research has focused on how the facial expressions of these six emotions are produced.[4,10,11] The consistent and differential facial muscle articulations (typically referred to as action units or AUs) associated with each emotion category have been identified.[12] An AU is a set of facial muscle articulations that results in a unique visible image feature. For example, when the frontalis (pars medialis) muscle contracts, there is a visible rising of the inner section of the eyebrows, and this is denoted AU 1.

The six emotion categories listed in the preceding paragraph are sometimes called “basic.”[13] By basic it is generally meant that emotions are discrete and have evolved to help us adapt to environmental challenges, ie, the response to a set of events is unique and common to everyone because it is an evolved behavior. With regard to facial expressions of emotion, this means that an automatic response to events of the same emotion category should result in a unique and consistent production (ie, the AUs used to produce that expression are the same across populations but differential between categories). Herein, we overview recent results from the study of these requirements. In particular, we wish to address the following question: Which, and how many, facial expressions of emotion are consistently and differentially produced by humans?

To date, most studies have focused on facial expressions defined by a single component emotion, eg, happy. We believe this is misguided for a variety of reasons. First, emotion categories need not be associated with a single feeling. While the feeling of joy can be defined as a component emotion (ie, happiness), the feeling of being joyfully surprised cannot. Experiencing a happily surprising event results in very different behaviors than those experienced when we are happy but not surprised. Similarly, happily surprised is different from happily disgusted, even though joy is a common denominator in these emotions. To properly address the question posed in the preceding paragraph (ie, which and how many facial expressions), we need to identify facial constructs that yield a consistent AU response pattern across populations and a differential response between categories, regardless of whether these are defined by single or multiple feelings.

What is an emotion then? In our examples above, happiness clearly defines a joyful state and, if of short duration, is considered an emotion. But what is happily disgusted? At first, this might seem contradictory, yet it is perfectly normal. Think about the behavior of someone after hearing a really funny but very disgusting joke (eg, some readers may be familiar with the 2005 film The Aristocrats). This compound feeling of joyfulness and repulsion is what we call a compound emotion.[14] Thus, compound emotions are also emotion categories. Both component and compound emotions are discrete entities that identify a unique cognitive state and associated behavior.

Key to understanding compound emotions is to note that these categories are as important as any other, ie, happily disgusted is a discrete category with a unique facial expression, as is the component emotion fear.

Herein, we review and expand on our previous studies of compound emotions. In particular, we introduce two new, previously undefined facial expressions of compound emotions, show results on spontaneous expressions, and illustrate the importance of intensity of AU activation to define and differentiate these emotion categories. Our results support the view that at least 17 compound emotions are associated with unique and differential facial expressions. We then provide a discussion of these results in the study, diagnosis, and treatment of psychopathologies.

Compound facial expressions of emotion

In our previous work,[14] we studied 15 compound facial expressions of emotion. We called them “compound” because these emotions are composed of two or more component categories. These 15 compound emotions are: happily surprised, happily disgusted, sadly fearful, sadly angry, sadly surprised, sadly disgusted, angrily fearful, fearfully surprised, fearfully disgusted, angrily surprised, angrily disgusted, disgustedly surprised, appalled, hatred, and awed. Two additional compound emotions that were not studied in our previous work, but that we will include in our analysis below are: happily fearful and happily sad

These discrete categories are a response to a set of common events, situations, or actions. Let us define each of them in more detail:

Happily surprised: This is an emotion we feel when we receive good or joyful unexpected news/outcomes: eg, when you applied for college, your favorite school accepted you even though you were convinced they would deny your application, or when a group of friends organize a surprise birthday party.

Happily disgusted: You feel joyful about an event, but repulsion regarding what it entails or what you see, hear, smell, or experience; eg, when someone tells you a very funny yet disgusting joke, or when you see someone making a funny but repulsive gesture.

Happily sad: You feel happy about an event that is tainted with sadness (ie, bittersweet); eg, when your children leave for college you feel happy and proud, but sad because they are leaving the nest.

Sadly fearful: A fearful event or action makes you sad; eg, you are threatened by someone emotionally close to you, or you are really disappointed when something nonthreatening is making you fearful.

Sadly angry: An event or action that makes you angry and leaves you disappointed or sad; eg, despite all your efforts to help, your son fails an important test.

Sadly surprised: A surprising event makes you sad; eg, your favorite soccer team is about to win a championship but loses to a last-minute goal.

Sadly disgusted: Something morally or sensorially distasteful that makes you sad; eg, when society or someone you care about disapproves of your lifestyle using vulgar language.

Fearfully angry: A fearful event or action makes you angry; eg, you are driving and you make a mistake that almost causes a serious accident, or you are told you might be fired because of something that you did not do.

Fearfully surprised: A surprising event makes you fearful; eg, while you are driving, a deer you had not seen crosses the road and almost causes a collision, or while hiking in the Rocky mountains you suddenly see a mountain lion 15 metres ahead.

Fearfully disgusted: A fearful event or action that is also morally or sensorially unpleasant; eg, an accident results in a life-threatening wound that is visually revolting.

Angrily surprised: A surprising event or action makes you angry; eg, unexpectedly and without provocation, a friend insults you.

Angrily disgusted: An event or action that makes you angry and is morally or sensorially unpleasant; eg, someone unjustly accuses you of doing something you did not do and that violates your moral code, or an elite athlete loses to a competitor using performance-enhancing drugs.

Disgustedly surprised: A surprising event or action that is morally or sensorially disgusting; eg, you are about to eat a piece of fruit that looks great but are surprised when a revolting odor reaches your nose.

Appalled: Horrified or angry and morally disgusted; eg, you are appalled at the incompetence of an expert or colleague.

Hatred: Extreme dislike or ill will; eg, racial or sexual hatred.

Awed: A feeling of reverential respect including the simultaneous feeling of fear and wonder; eg, you look in awe at the vastness and the unknown in the jungle at night.

Furthermore, the facial expressions of these emotion categories might have evolved to help us better solve specific behavioral or social problems.[9] For example, one hypothesis is that disgust is a mechanism of germ avoidance.[15] Similarly, compound emotions such as awe might serve as a social glue in large groups of people through communal events, such as those involving the arts, natural phenomena, and religion.[16]

Two things need to be clarified here. First, as can be seen from the list of emotions given above, some compound emotions are identified by a single word in English. However, these categories are still defined as compounds because they include two emotional feelings. For example, hateful and appalled are defined as the compound feeling of anger and disgust, with the emphasis on anger when hateful and on disgust when appalled. Awe is the compound feeling of fear and wonder (surprise), with the emphasis on the latter (eg, horripilation in awe is believed to be a residual behavior of fear). The fact that these compound emotions are defined as single words does not mean they are more or less relevant than the others. This is a characteristic of the English language and may not be the same in other languages. For example, in Albanian, awe is described as an abrupt or sudden fear and, hence, requires multiple words to define it. The same holds true for compound words: not all languages use two or more words to define them. For example, in Mandarin Chinese, happily surprised is written as a single word. Finally, emotions associated with a single feeling may be named using one or more words. For instance, in Igbo (a language spoken in the southeastern region of Nigeria), emotions such as happy are generally described using multiple words. It is important to note that cognitive categories need not match language; that is, a compound emotion may be given by a single word in one's language, while a component emotion may be described with multiple words. The important difference is in the simultaneous feelings one experiences, not how the emotion category is written. When we experience a single feeling (eg, joy), we will talk about component emotions (eg, happy). When we experience multiple feelings (eg, joy and surprise), we will talk about compound emotions (eg, happily surprised).

As mentioned in the Introduction of this paper, an emotion that is externalized should have a facial expression that is produced with the same AU combination by people of different cultures but differential from those AU patterns employed to display other emotion categories. We thus wish to know whether compound emotions are displayed with unique and differential facial expressions. In other words, when, in a situation that yields a compound emotion, is human behavior consistent and differential?

Consistent and differential action units

We denote the six emotion categories studied in the past (ie, happiness, surprise, anger, sadness, disgust, and fear) component emotions. The other 17 emotion categories introduced in the preceding section are denoted as compound emotions.

The very first thing we wish to know is if these 17 compound emotions yield facial expressions that are consistent across people but differential between categories. This had already been shown to be the case for component emotions,[10,12] but not for compounds. To study this, we analyzed the facial expressions of compound emotions made by people of different cultures.[14] We then manually identified the AUs that were employed to express each emotion category. Specifically, we identified the AUs that are consistently used by a large majority of people (>70%). Table I summarizes these results. As can be seen in this table, all categories yield a unique facial expression. That is, the AU pattern of activation is distinct amongst emotion categories. The most similar categories are angrily disgusted, hatred, and appalled. This is to be expected, because these three emotion categories include the same component categories — the simultaneous feeling of anger and disgust. The difference is in the relevance of these components within the compound. In angrily disgusted, we feel equally angry and disgusted, whereas in hatred the emphasis is on anger and in appalled the emphasis is on disgust. As can be seen in Table I, this results in slightly different expressions. In angrily disgusted there is the consistent use of three AUs (4, 10, and 17) with AUs 7 and 9 playing a secondary role. In contrast, hatred mainly uses AUs 4 and 10, with 7 and 17 playing the secondary role. And, finally, in appalled, people primarily use AUs 4 and 10, plus the likely addition of 9 and/or 17.

Another way that compound categories of closely related emotional states are differentiated is in the intensity of the activation of their AUs. As mentioned earlier in this paper, AUs correspond to visual image changes caused by the activation of a specific set of facial muscles. This image change can, however, be more or less visible. When the contraction of the associated muscles is stronger, we generally observe a large image change. We thus refer to this as the intensity of the AU. Typically, we use five values to denote the intensity of activation of an AU. They are denoted, from smallest to largest intensity, a through e. shows the percentage of times people use a given intensity (or larger) in each AU for each of the 23 emotion categories.

As is made clear by this figure, the AU patterns of activation used by people to express each emotion are not only different; the intensities of AU activation are also distinct. This is important because it further illustrates the many ways in which the expressions of different discrete emotion categories are consistently produced by people of different cultures.

In Figure 3, we also see that the AUs used to express a compound emotion are consistent with the AUs used to express its component (subordinate) categories. That is, the resulting AU pattern is unique to that compound emotion (a fact that allows us to differentiate this facial expression from the others), but the resulting expression is also consistent with the AU patterns used to display its component categories. Hence, feeling happily surprised is consistent with the state of happiness and surprise. This obviously makes these expressions easier to be visually recognized by an observer. An important difference to note is the intensity of activation of each AU. Note how some AUs are emphasized more in the compound than in the subordinate categories (ie, intensity of production is increased), whereas others are deemphasized. This difference in production further facilitates the visual distinction of each facial expression while leaving the active set of AUs consistent across compound and subordinate categories.

Behavioral experiments by our group[17] show that people are quite good at visually discriminating these facial expressions of emotion. Our studies have tested people's ability to name and visually discriminate these expressions of emotion. For instance, in a cognitively taxing task, participants were shown an image of one of the component or compound facial expressions and given a list of 21 possible labels (ie, happy, happily surprised, sad, sadly angry, etc). Westerners and East Asian participants were able to correctly name 19 of the emotion categories introduced in the present paper. Most importantly, the results in these two groups (Westerners and East Asians) were statistically identical. And, notably, a set of eight emotion categories was recognized more readily than the rest by both groups. In another experiment, participants were instructed to indicate by key press whether the two images shown on the screen belonged to the same emotion category or not. We obtained the same result. All emotion categories were visually discriminated, but eight of them were not only discriminated more accurately, but more rapidly too. These results suggest there are at least two groups of facial expressions of emotion. One group is hypothesized to play a major role in cognitive and/or social tasks, and are thus more readily recognized. The other expressions are also universally recognized by people of different cultures, but seem to play a less central role in our daily cognitive and social tasks. An alternative hypothesis is that evolution has selected these few emotions because they provided some survival advantage. Our group is currently working on trying to better understand these differences and its implications in cognition, behavior, and psychopathologies.

Spontaneous facial expressions

Thus far, we have presented studies of facial expressions of emotion collected in the laboratory. Next, we wish to know if spontaneous expressions are produced with the same AUs identified in the laboratory. This is important because the neural systems involved in the production of posed and spontaneous expressions are believed to differ. To answer this question, we searched the Web for images of spontaneous facial expressions. Our search was based on keywords (eg, happy, happily disgusted, happy and disgust, etc). We searched newspaper websites (eg, The New York Times. The Guardian) and television channel sites (eg, CNN, CBS news) and also used Google® Images to identify these expressions.

Our search started with a qualitative analysis. When an image was visually identified as appearing to express one of the 17 compound emotions listed above by one of the two authors, we proceeded to quantitatively evaluate the expression by manually annotating its AUs. We found that the AU pattern of activation of spontaneous expressions is identical to those seen in the laboratory. This is not a surprising finding for a couple of reasons. First, humans are extremely good at producing facial expressions of emotion. We are so accustomed to interacting with others nonverbally that replicating this in the laboratory does not pose a major problem. (Surprisingly, happiness might actually be the one major exception to this rule. [18]) Second, when collecting images in the laboratory, we were very careful to instruct the subjects on how to provide as naturalistic an expression as possible. Participants were first given a few scenarios where an emotion is experienced. Then, participants had the opportunity to practice in front of a mirror to see if their expression matched what they intended before pictures were taken. This procedure yielded the same results we have now observed in spontaneous expressions, further validating the results summarized in Table I and Figure 3.

Clinical applications

Emotions play a major role in human health. Most people have felt upbeat or beaten up emotionally at some point in their lives. When these occurrences are transitory (hours or a few days), they change our mood. If these moods linger, they can become problematic and even chronic. Involuntary long-lasting effects may develop in psychopathologies, for example, clinical depression or post-traumatic stress disorder (PTSD), with genetics believed to be a contributing factor in this transition.[19] Other psychopathologies are believed to be developmental, as is the case in autism spectrum disorder (ASD). Understanding how many facial expressions of emotion neurotypicals produce and recognize is essential to further our study of these disorders, as well as for the development of coping mechanisms and treatment options.

The very first hurdle that needs to be cleared in these cases is to provide a correct and accurate diagnosis. This has proven difficult, mainly because of the heterogeneity and reification of psychopathologies. For instance, PTSD is generally diagnosed after lengthy interviews with a knowledgeable psychiatrist who usually administers the Clinician-Administered PTSD Scale (CAPS). CAPS diagnosis is based on the presence of several symptoms described in the Diagnostic and Statistical Manual of Mental Disorders (DSM) rather than pathophysiologically. To be able to resolve this issue, we need to better understand the differences in production and visual recognition of facial expressions of emotion.

Our general hypothesis is that pathophysiological distinctive disorders result in a different production and/ or visual interpretation of facial expressions of emotion. Before we can fully address this hypothesis though, we need to identify all component and compound facial expressions of emotion. To date, we have identified 23 facial expressions of emotion, but there could be more. Some researchers[20] have suggested that contempt is a component emotion. If this were confirmed, it might be possible to compound contempt with other component categories such as happiness or disgust, to yield categories such as derision.

For the time being, we believe physicians and medical personnel should learn to quickly identify the known 23 facial expressions of emotion in patients. As we have seen above, we are all quite good at visually identifying these facial constructs, but we believe medical professionals should be proficient at this task. To learn to identify these expressions, one can pay close attention to the results given in the table and figures above. Our research laboratory at The Ohio State University is also developing an interactive (Web-based) application to train medical personnel to become better at visually recognizing these expressions. People unconsciously externalize their internally felt emotions through facial expressions. Identifying these in patients could prove as valuable as listening closely to what they have to say.

Our research group is also developing computer algorithms that can recognize these facial expressions automatically.[21] This technology would free the medical professional from having to analyze the face and would eliminate or, at least facilitate, training. All that will be needed is a camera and computer, and the system will provide real-time information to the physician. Our current system recognizes expressions of emotion with >75% accuracy, which is better than untrained individuals can do.[14] Improvements on this technology are sure to revolutionize clinical practice in the next few years. Advances in face detection, for instance, are fueling improvements in the recognition of facial expressions.[22,23]

.

Another likely difference between the production of facial expressions of emotion in neurotypicals and individuals with psychopathologies is variations in the intensity of activation of AUs. For example, are AUs in sadness (including its compounds, eg, sadly angry, sadly surprised, etc) displayed with larger or smaller AU intensity in clinical depression? Characterizing these differences will allow us to develop protocols for evaluating the automatic annotations given by the computer software outlined in the preceding paragraph.

.

The above covers the production of facial expressions of emotion. The visual recognition of these expressions is also expected to be atypical in psychopathologies. Past research has shown differences in the visual recognition of facial expressions of component emotions,[24-27] but there is not yet any research that has studied the perception of compound emotions in the clinical population. This is likely to be a productive area of research in clinical and translational medicine.

The set of six emotions used in the research studies listed above is likely to be insufficient to describe all psychopathologies defined in the DSM. This is especially true given the poor heterogeneity and reification of psychopathologies. There is limited variability that can be readily and consistently observed in the facial expressions of joy, surprise, sadness, anger, disgust, and fear across psychopathologies. By studying a much larger number of emotions (ie, variables), it is much more likely to find common patterns of production across psychopathologies. For example, clinical depression might result in an increased production of compound expressions with a sad component (eg, sadly angry, sadly fearful, sadly disgusted), even in the absence of additional facial expressions of sadness. Alternatively, the intensities of AU production might be diminished in all compounds. Basic research is needed to study this, but psychiatrists and other medical professionals should also report what is observed in their practice. These observations can prove invaluable to researchers, and can serve as hypotheses for future studies. This would require that medical professionals become proficient in the visual interpretation of all facial expressions of emotion.

Discussion

Compound emotions are typically observed emotions in everyday life. When riding a rollercoaster we rarely feel just happy or fearful. Typically, people feel happily fearful. This may seem contradictory at first because happiness and fear seem polar opposites, yet these are common compound emotions people experience. Our research shows that the facial expressions that accompany these internally felt emotions are consistent across people and differential between emotion categories. Furthermore, the associated facial constructs seem to be universally visually recognized by observers, even in challenging conditions. In our previous work we defined 21 compound facial expressions of emotion.[14] Herein, we have further extended this to 23 (Table I; Figures 1, 2).

Much still needs to be done to fully understand compound emotions and their facial expressions, however. For instance, we do not yet know how many emotion categories there are. Specifically, if contempt (or others) were found to also be consistently produced by people of distinct cultures, then many more compound categories would be possible. Furthermore, it might be possible to have more than two simultaneous feelings. For example, it is likely that in some circumstances one could feel fearful, surprised and happy, eg, while riding in one of the attractions in an amusement park, it is common to experience surprises that make us fearful and happy. But it is unknown whether some of these compounds result in a consistent and differential facial expression.

Also, our results show that, while almost all the facial expressions of emotion discussed above are visually recognized by people, there is a small set of expressions we seem to be better at recognizing.[17] Current results suggest this distinction is biological, since people of different cultural upbringings show the same recognition ability. But we do not yet know why this difference exists, or what its implications are.

The open questions discussed above are of high importance in understanding human cognition and behavior and are essential to further our understanding of psychopathologies. Nevertheless, there is much that can already be achieved today. For example, as already mentioned, we believe that medical professionals should familiarize themselves with these expressions and their meanings. Ideally, physicians working with populations at risk (eg, psychiatrists, medical doctors working in the emergency room) would become proficient in recognizing these emotions. This would provide much-needed information, not only to diagnose, but also to detect potential risks (eg, risk of suicide, depression, PTSD). We are currently working on the design of a Web-based training system for professionals that will help fill in this gap. A related priority of our research group is to develop computer vision algorithms that can recognize these facial expressions of emotion automatically, so even untrained professionals can detect potential problems. It will also be necessary for medical doctors and researchers to develop protocols on how to interpret and respond to these observed behaviors.

Tableau I

Prototypical action units (AUs) of the 17 compound emotions described in this paper. Each row in the table lists the AUs used to produce the facial expression of each emotion category. The percentage of people using each AU is specified in parentheses, right of the AU number. The results of the first 15 emotion categories were derived from a total of 230 sample images per category, while those of the last two emotions were obtained from 20 sample faces per category.

Category	Prototypical AUs.
Happily suprised	1 (95%), 12 (100%), 25 (100%)
Happily disguted	10 (98%), 12 (100%), 25 (100%)
Sadly fearful	1 (86%), 4 (94%), 20 (70%), 25 (97%)
Sadly angry	4 (97%), 15 (83%)
Sadly surprised	1 (84%), 4 (90%), 25 (99%), 26 (90%)
Sadly disgusted	4 (97%), 10 (85%)
Fearfully angry	4 (99%), 20 (84%), 25 (98%)
Fearfully surprised	1 (93%), 2 (80%), 5 (74%), 20 (90%), 25 (99%)
Fearfully disgusted	1 (77%), 4 (75%), 10 (92%), 20 (88%), 25 (98%)
Angrily surprised	4 (99%), 25 (100%), 26 (94%)
Angrily disgusted	4 (98%), 10 (93%), 17 (79%)
Disgustedly surprised	1 (93%), 2 (90%), 5 (73%), 10 (91%)
Appalled	4 (85%), 10 (94%)
Hateful	4 (92%), 10 (86%)
Awed	1 (95%), 2 (89%), 5 (77%), 25 (97%)
Happily fearful	1 (90%), 2 (85%), 12 (100%), 25 (100%), 26 (100%)
Happily sad	4 (85%), 6 (75%), 12 (100%), 25 (75%)