The Complex Emotion Expression Database: A validated stimulus set of trained actors.

The vast majority of empirical work investigating the mechanisms supporting the perception and recognition of facial expressions is focused on basic expressions. Less is known about the underlying mechanisms supporting the processing of complex expressions, which provide signals about emotions related to more nuanced social behavior and inner thoughts. Here, we introduce the Complex Emotion Expression Database (CEED), a digital stimulus set of 243 basic and 237 complex emotional facial expressions. The stimuli represent six basic expressions (angry, disgusted, fearful, happy, sad, and surprised) and nine complex expressions (affectionate, attracted, betrayed, brokenhearted, contemptuous, desirous, flirtatious, jealous, and lovesick) that were posed by Black and White formally trained, young adult actors. All images were validated by a minimum of 50 adults in a 4-alternative forced choice task. Only images for which ≥ 50% of raters endorsed the correct emotion label were included in the final database. This database will be an excellent resource for researchers interested in studying the developmental, behavioral, and neural mechanisms supporting the perception and recognition of complex emotion expressions.

The ability to visually perceive, interpret, and categorize emotional expressions from the face is a central component of social communication. Even in infancy, humans perceive and use facial expressions as social signals (e.g., [1]). By adulthood, we are experts at using emotional expressions to predict and guide behavior.

The vast majority of empirical work investigating the developmental, behavioral, and neural mechanisms supporting the perception and recognition of facial expressions is focused on basic expressions (e.g., [2-9]). These expressions provide signals about a specific set of universal emotions, including: anger, disgust, fear, happiness, sadness, and surprise [10]. Much less is known about the underlying mechanisms supporting the perception and identification of non-basic, complex expressions, which provide signals about emotions related to more nuanced social behavior and inner thoughts [11]. The developmental, behavioral, and neural mechanisms supporting the perception and recognition of complex expressions could be similar, different, or overlapping. There is some existing work suggesting that the developmental and behavioral mechanisms may be different [see 12–13].

To facilitate more research on the processing of complex emotion expressions, we developed and validated a database of digital photographs showing young adult actors making complex expressions. Here, we discus categories of complex expressions and explain why the database primarily consists of basic and complex social sexual expressions [12]. Next, we review limitations of existing databases that do feature complex expressions and show how this new database fills some of those gaps. Finally, we identify the primary goals and strategies motivating the development of this database as well as the validation procedure and results.

Categories of complex emotional expressions

Based on the notion that expressions provide adaptive social signals that enable rapid appraisal and preparation to act [14], several investigators have suggested there are subcategories of complex expressions based on their signaling properties [11, 12, 15]. For example, there is precedent for distinguishing complex cognitive and complex social expressions (for review see [12]). Briefly, complex cognitive expressions (e.g., pensive) reflect inner thoughts (i.e., they do not necessarily result from interactions with people) and have low valence and arousal, while complex social expressions are elicited in specific social contexts and vary in arousal and valence (e.g., serious) [15]. Recently, we proposed additional organizational structure within the category of complex social expressions to include sub-categories, like social self-conscious (e.g., guilt, pride) and social sexual (e.g., desire, flirtatious) expressions [12]. We argued that these sub-categories reflect additional functional segregation of the signaling properties of the expressions. Self-conscious expressions facilitate behavioral adherence to moral standards and are evoked by self-reflection and self-evaluation. In contrast, social sexual expressions provide signals about the status of romantic and sexual relationships. For example, the eyebrow flash together with a smile is a signal of sexual interest [16].

Existing emotional expression databases

There is a plethora of digital stimuli databases for researchers to access for researching questions about the perception and recognition of basic expressions. These databases largely include static photographs and vary by age of the expresser to include infants, children, adolescents, or adults [17, 18, 19, 20]. Most of the databases include both male and female expressers and some have racial/ethnic diversity (e.g., [21, 22, 23]). In contrast, there are many fewer stimulus database options for researchers interested in investigating the perception and/or recognition of complex expressions.

We have summarized all of the available stimulus databases that include some form of complex emotional expression in Table 1. What is evident is that when complex expressions are included in a stimulus database, there are not many categories of complex expressions to draw upon for study as a researcher. For example, several databases only include the single complex expression of contempt [21, 29, 36, 38]. This expression is not representative of all complex expressions and may not even represent a broad category of complex expressions very well. As a result, researchers cannot learn much about the perception and recognition of complex expressions more generally by investigating responses to this single expression.

Table 1

Existing emotion expression stimulus databases that include complex expressions.

Database	Paper	Complex Expressions	Actors as Models	Age of Models	Number of Models	Ethnicity of Models
ADFES	[24]	contempt, embarrassment, pride	No	18–25 yrs	20 + 2 authors	White
BINED	[25]	frustration, amusement	No	“Adults”	256	White, Latinx
BP-4D Spontaneous	[26]	embarrassment, pain	No	18–29 yrs	41	Asian, Black, Hispanic, White
CAM Face-Voice Battery	[27]	appalled, appealing, confronted, distaste, empathic, exonerated, grave, guarded, insincere, intimate, lured, mortified, nostalgic, reassured, resentful, stern, subdued, subservient, uneasy, vibrant	Yes	“Adults”	Not provided	Not provided
Chicago	[28]	threatening	Some	18–40 yrs	158	Black, White
CK +	[29]	contempt	Not provided	18–50 yrs	210	Black, White, other
Dartmouth	[18]	content	No	6–16 yrs	80	White
DuckEES	[19]	pride, embarrassment	Yes	8–18 yrs	36	White, other
DynEmo	[30]	amusement, interest, irritation, worry	No	“undergraduates”	43	Not provided
Eu-Emotion	[20]	ashamed, bored, disappointed, excited, frustrated, hurt, interested, jealous, joking, kind, proud, sneaky, unfriendly, worried	Yes	10–70 yrs	19	White, Black, Mixed
GEMEP-CS	[31]	amusement, anxiety, cold anger (irritation), despair, hot anger (rage), fear (panic), interest, joy (elation), pleasure (sensory), pride, relief, admiration, contempt, tenderness	Yes	25–57 yrs	10	Not provided
JACFEE	[21]	contempt	Not provided	Not provided	56	Asian, White
McEwan Faces	[32]	compassionate, critical	Yes	“young actors,” “mature actors”	31	White, Black, Asian
McGill Face Database	[33]	92 expressions: affectionate, comforting, contemplative, desire, distrustful, embarrassed, flirtatious, friendly, imploring, jealous, playful, and sympathetic	Yes	23–29 yrs	2	White
MPI	[34]	agree, aha, arrogance, bored, annoyed, confused, contempt, don't care, didn't hear, disagree, disbelief, don't know, don't understand, embarrassment, evasive, imagine, impressed, insecurity, compassion, not convinced, pain, annoyed, thinking, smiling, tired, doe-eyed	No	20–30 yrs	19	Not provided
MSFDE	[35]	embarrassment, shame	Yes	“Young adults”	24	Asian, White, Black
Radboud	[36]	contempt	No	“Children,” “Adults”	49	White
STOIC	[37]	pain	Yes	20–45 yrs	10	Not provided
TFEID	[38]	contempt	Not provided	Not provided	40	Not provided
UT Dallas	[39]	boredom, disbelief, laughter, puzzlement	No	18–41 yrs	284	White, Black, Asian, Hispanic, Other

Across these databases, the most frequently represented complex expressions are social self-conscious expressions, including pride and/or embarrassment [19, 20, 24, 26, 30, 35]. These stimuli are particularly useful for asking questions related to the perception and recognition of signals about adherence to moral standards that involve self-reflection and self-evaluation. For example, they would be particularly useful for addressing questions about clinical or at-risk populations that exhibit impairments in these abilities or about the developmental emergence of sensitivity to these expressions. However, if researchers are interested in other kinds of questions like whether there are age-related changes in sensitivity to social sexual expressions or whether sensitivity to these expressions changes as a function of pubertal development or relationship status (i.e., single, committed romantic relationship), there are limited options.

The CAM Face-Voice Battery includes the expressions of appealing, empathetic, and intimate [27]. The GEMEP database includes the complex social expressions of tenderness, interest, and pleasure [31]. The McGill Face Database [33] is composed of 93 different expressions that include basic, complex cognitive, and complex social expressions. While the McGill Database has an excellent range of complex expression stimuli, the limitation is that there are only two actors (one male, one female) who portray all the expressions, which restricts the generalizability of any findings. Specifically, there is a confound between the actor and the expression such that it is impossible to discern whether differential responses are specific to the actor, expression, or the interaction between the two. It is important to have multiple actors portraying the same expression to prevent this confound. Finally, if researchers cobble together stimuli across multiple databases in a single experiment, they risk including methodological differences that were introduced in the creation of the stimuli across expressions as an additional confound. Given the limitations in the existing databases, there is a clear need for complex expression stimuli, especially those that portray expressions from the multiple sub-categories of expressions.

Complex Emotion Expression Database (CEED)

Here, we introduce a database of facial emotional expressions that we developed to address some of these limitations. The database includes both basic and complex social expressions portrayed by young adult actors with formal training and extensive performance experience. In particular, the complex social expressions represent the subcategory of social sexual expressions, including affectionate, attracted, betrayed, brokenhearted, contemptuous, desirous, flirtatious, jealous, and lovesick. We were especially interested in creating a database of these expressions because they are underrepresented in the literature and they are central to hypotheses that we are exploring in our own work. Previously, we predicted that emerging perceptual sensitivity to different subcategories of complex expressions may follow different developmental trajectories based on the relevance of each for accomplishing social developmental tasks of childhood and adolescence [12]. For example, we predicted that sensitivity to complex social sexual expressions would only emerge in adolescence and as a function of pubertal development as adolescents begin to explore and participate in romantic and sexual partnerships with peers, which is a primary social developmental task of adolescence [12, 40]. We are using these stimuli to address these questions.

The Complex Emotion Expression Database (CEED) includes 480 images of eight young adult actors creating six basic and nine complex social sexual emotional expressions. The actors are both female and male with some racial diversity. The images were independently rated by nearly 800 participants to validate the perception of the expression.

Method

Participants

Models

Eight professional actors were hired to pose basic and complex facial emotional expressions. The actors were young adults who ranged in age from 18–27 years (M = 20.9 years, SD = 3.1), and included four Black and four White individuals, and four males and four females (see Table 2). All of the actors had formal training in acting. They were all in either undergraduate or graduate level theater training programs and had extensive performance histories. The actors all provided written consent for their photos to be taken and used for research purposes. The individuals pictured in Figs 1 and 2 of this manuscript have provided written informed consent (as outlined in PLOS consent form) to publish their image alongside the manuscript.

Table 2

Actor demographics.

Actor ID	Age (Years)	Gender	Ethnicity	# of Images
20yoWM1	20	M	White	81
18yoWM2	18	M	White	71
27yoBM1	27	M	Black	68
19yoBM2	19	M	Black	58
19yoWF1	19	F	White	74
21yoWF2	21	F	White	15
24yoBF1	24	F	Black	88
19yoBF2	19	F	Black	25
Mean (SD)	20.9 (3.1)			60.0 (26.3)

Fig 1

Highly rated basic expression images in CEED.

Ratings were measured in a 4AFC task and computed as endorsement scores. The scores reflect the percent of total raters who endorsed the target label for the expression (i.e., picked the label “angry” for an angry expression). Only images with endorsement scores ≥ 50% are included in the database. These images have scores that range from 87–94.5%.

Fig 2

Highly rated complex expression images in CEED.

Ratings were measured in a 4AFC task and computed as endorsement scores. The scores reflect the percent of total raters who endorsed the target label for the expression (i.e., picked the label “jealous” for a jealous expression). Only images with endorsement scores ≥ 50% are included in the database. These images have scores that range from 64.5–81.5%.

Raters

A total of 870 people rated the images. However, it is possible that these are not all unique individuals because we did not preclude raters from participating in the multiple versions of the task that were available on Amazon’s Mechanical Turk (MTurk). The raters ranged in age from 18–82 years old (M = 34.5 years, SD = 11.4). The self-reported gender-identity of the sample included 449 males, 416 females, 1 gender non-conforming individual, and 4 individuals who did not report a gender identity. We did not ask raters to report their racial or ethnic identity. Participants provided implied consent online before they rated the stimuli. These procedures were approved by the Institutional Review Board of Penn State University.

Procedure

Photographing the expressions

Actors were invited to the Laboratory of Developmental Neuroscience to be photographed in individual sessions. Photographs were taken using a Fujifilm FinePix S4200 digital camera with a 24 mm lens. The photographs were taken in a quiet room with the actors sitting on a chair in front of a white wall. Lighting was standardized across sessions, which included overhead fluorescent lights, an umbrella reflector, and an 85-watt spotlight on the actor’s face. The camera was stabilized on a tripod approximately five feet away from the actor. The zoom function was used to focus in on the actor’s shoulders, neck, and head.

Images were acquired of six basic (angry, disgusted, fearful, happy, sad, surprised) and nine socially complex expressions (affectionate, attracted, betrayed, brokenhearted, contemptuous, desirous, flirtatious, jealous, lovesick). Emotional expressions were photographed in separate blocks (i.e., all angry expressions, all contempt expressions). Prior to photographing an expression, the researchers provided the actors with a definition and image of the target emotional expression. The actors were instructed to express the specific emotional expression using a method acting approach. The researchers also provided a series of example scenarios that would elicit the emotional expression. The researchers gave the actors time to invoke the emotion and asked the actors to let the researcher know when they were ready to generate the emotional expression to be photographed. The actors produced the expression as the researcher took multiple photographs of the actor.

Image processing

Images that were blurry or that were determined by the researchers not to express the target emotion were excluded from further processing. Images that were selected for additional validation were cropped in Photoshop to remove much of the background scene. This procedure was not fully standardized so some of the images were cropped to only include the head and neck of the actor. Other images include the shoulders of the actor and some of the wall behind the actor. All of the images are greyscaled and standardized to a resolution of 300 dpi and a width of 4 inches. The length of the images varies slightly.

Validating the expression stimuli

To measure the extent to which the actor was successfully able to produce the emotional expression they were asked to express, the images were externally rated. Participants were recruited from MTurk to rate the expressions of the images. This was executed in five versions of the validation task on MTurk, with each iteration including different images. As a result, different participants rated different photographs. However, we did not limit rater participation to just one of the validation tasks, so it is possible that some participants rated multiple images. A total of 719 images (296 basic and 423 complex) were rated on MTurk. Every image was rated by at least 50 people (number of raters per image M = 100.1, SD = 84.7).

The validation task was a 4-alternative forced choice task (4AFC). Raters were presented with a single image and four emotion expression labels simultaneously. They were instructed to pick the best label that described the expressed emotion. Raters had an unlimited amount of time to respond and the image stayed on the screen until the rater provided a response. The order of the images was randomized within each task for each participant.

Critically, the alternative label choices were systematically controlled. First, the alternative label options were always within valence, which prevented participants from easily eliminating potential labels on the basis of valence. For example, if a participant was shown a negatively valenced emotion expression (e.g., angry), then all the labels described negatively valenced expressions (e.g., angry, disgusted, arrogant, sarcastic). Second, the labels always included descriptions of both basic and complex emotion expressions. This was essential to identify the specificity of the perception of the expression. For the images featuring basic expressions, the labels included the target expression (e.g., angry), an alternative basic expression (e.g., disgust), and two alternative complex expressions (e.g., arrogant, sarcastic). Similarly, for the images featuring complex expressions, the labels included the target expression (e.g., jealous), an alternative complex expression (e.g., despondent), and two alternative basic expressions (e.g., sad, fearful). Third, the labels describing complex expressions were selected from the labels used in the Revised Reading the Mind in the Eyes Test (RMET) [41]. The RMET is a commonly used measure of emotion expression perception that includes complex expressions and presents participants with a similar 4AFC paradigm. The answer choices for each expression are provided in S1 Table.

Data analysis

The primary dependent variable was accuracy, the ability to identify the target label for the emotional expression represented in the image. Raters did not have to provide complete data to be included in the analyses. We included partial data in the analyses. Missing data were not interpolated. To identify unengaged participants, we excluded those who scored at or below chance level performance. To do so, we computed total mean accuracy across all trials for each participant. Raters whose total accuracy was ≤ 25% were excluded from the analysis (n = 74).

To determine the external validity of the emotional expression represented in each image, we computed endorsement scores. The endorsement score was defined as the percent of total raters who endorsed the target label for the expression (i.e., picked the label “angry” for an angry expression). This is essentially an accuracy score if one considers the target label the “correct” label. Therefore, the frequency with which participants picked an alternative label reflects the “error” on a given trial.

The endorsement scores for all images in the database are provided in S2 Table. We used the endorsement scores to further down select the items in the database so that only items with good external validity are provided. We include all images with endorsement scores ≥ 50%, which indicates that an absolute minimum of 25 people (50% of 50 raters) endorsed the expression displayed in the image to be represented by the target label when presented with three alternative labels that were of similar valence. In addition, we analyzed the error on each item to identify the proportion of raters who chose each of the alternative labels. For example, researchers can determine whether raters are likely to misidentify an expression (e.g., anger) consistently with the label for another expression (e.g., betrayed) or not. Together with the endorsement scores, this error information helps to define the relative specificity of these scores for each stimulus.

Results

The final set of raters who provided data for the analyses of the endorsement scores of the expression images included 796 individuals. They ranged in age from 18–82 years old (M = 34.8 years, SD = 11.6). The self-reported gender-identity of the raters included 403 males, 388 females, 1 individual who identified as gender non-conforming, and 4 individuals whose gender was not provided.

Participants rated a total of 719 images. From these images, a total of 480 (243 basic and 237 complex) images (66.8%) elicited endorsement scores ≥ 50% and are included in the database. In S2 Table, the images are cataloged with endorsement scores as well as information about the raters (number, gender) and the frequency of responses to the alternative labels.

Endorsement scores for images representing basic expressions

Among the 243 images representing basic emotion expressions in the database, 43 are of angry, 26 are of disgusted, 48 are of fearful, 49 are of happy, 33 are of sad, and 44 are of surprised expressions. Fig 1 illustrates highly endorsed images representing each of these six basic expressions. Endorsement scores for images representing basic expressions ranged from 50.0–95.3% (M = 75.0%, SD = 11.0%) (see Table 3). The images representing surprise expressions generated the highest endorsement score (M = 79.3%, SD = 12.8%) and the images representing disgusted expressions generated the lowest endorsement score (M = 70.6%, SD = 9.5%). All eight actors contributed images representing the expressions of fearful, happy, and surprise. Seven actors contributed images representing the expression of anger and six actors contributed images representing the expressions of disgusted and sad.

Table 3

Endorsement scores for basic images.

Expression	# of Images	Mean	Range	Alternative Basic	Alternative Complex
Angry	43	76.6 (10.8)	52.0–93.0	6.5 (7.9)	8.5 (5.9)
Disgusted	26	70.6 (9.5)	51.8–87.0	8.0 (6.6)	10.7 (5.4)
Fearful	48	76.5 (10.6)	52.4–93.0	3.5 (2.5)	10.0 (6.6)
Happy	49	71.2 (9.8)	50.8–88.3	3.9 (3.6)	12.4 (10.3)
Sad	33	74.1 (10.0)	53.5–90.6	5.2 (4.3)	10.4 (8.0)
Surprised	44	79.3 (12.8)	50.0–95.3	11.7 (12.3)	4.5 (3.5)
Basic	243	75.0 (11.0)	50.0–95.3	6.3 (7.5)	9.4 (7.5)

Contents represent Mean (SD). Images are included in final dataset if mean endorsement score ≥ 50%.

These results reveal consistent endorsement of the target label for the expressions in the images. In addition, there was also specificity in the perception of the expressions. For example, across all the images in which basic expressions were exhibited, the alternative basic (M = 6.3%, SD = 7.5%) and alternative complex (M = 9.4%, SD = 7.5%) labels were rarely endorsed (see Table 3). The only set of images in which there was some willingness to endorse an alternative label was for the expression of surprise. On average 11.7% (SD = 12.3) of raters chose the alternative basic label “happy.” Also, it is important to note that although the basic expression labels were repeated more frequently than were alternative labels (because there are fewer to choose from within valence), this did not bias raters’ selection of alternative labels. S2 Table shows that for all basic expressions, the most common alternative label picked was a complex (not basic) expression label. Similarly, for complex expressions, the most common alternative label picked was also a complex label, even though there were two basic label options.

Endorsement scores for images representing complex expressions

Among the 237 images representing complex expressions in the database, 36 are of affectionate, 19 are of attracted, 20 are of betrayed, 36 are of brokenhearted, 19 are of contemptuous, 46 are of desirous, 22 are of flirtatious, 9 are of jealous, and 30 are of lovesick expressions. Fig 2 illustrates highly endorsed images representing each of these complex expressions. Endorsement scores for images representing complex expressions ranged from 50.0%– 81.5% (M = 62.4%, SD = 7.7%) (see Table 4). The images representing the expression of desirous generated the highest endorsement score (M = 65.3%, SD = 9.2%), whereas images representing the expression of betrayed generated the lowest endorsement score (M = 57.2%, SD = 5.7%). All eight actors contributed images representing the expressions of desirous and brokenhearted. Seven actors contributed images representing the expression of flirtatious; six actors contributed images representing the expressions of attracted, betrayed, and lovesick; five actors contributed images representing the expression of contemptuous; and three actors contributed images representing the expression of jealousy.

Table 4

Endorsement scores for complex images.

Expression	# of Images	Mean Score	Range	Alternative Complex	Alternative Basic
Affectionatea	36	62.0 (6.5)	50.8–75.5	16.0 (8.1)	6.1 (3.0)
Attracted	19	63.9 (8.2)	50.0–81.5	17.3 (8.0)	9.4 (8.9)
Betrayed	20	57.2 (5.7)	50.0–64.5	16.6 (6.1)	13.1 (6.1)
Brokenhearted	36	63.5 (7.9)	50.0–81.2	19.7 (7.8)	8.4 (6.4)
Contemptuous	19	61.9 (7.0)	51.6–75.4	17.3 (7.3)	10.4 (7.7)
Desirousa	46	65.3 (9.2)	50.8–79.6	15.3 (10.0)	4.2 (2.8)
Flirtatious	22	62.0 (7.3)	50.9–74.4	12.8 (7.1)	12.6 (11.6)
Jealous	9	60.3 (4.9)	51.5–67.9	25.5 (4.1)	7.1 (4.5)
Lovesick	30	60.6 (6.5)	50.9–77.4	8.0 (3.6)	15.7 (8.4)
Complex	237	62.4 (7.7)	50.0–81.5	15.7 (8.7)	10.0 (8.1)

Contents represent Mean (SD). Images are included in final dataset if mean endorsement score ≥ 50%.

aRaters were shown one alternative basic label and two alternative complex labels in addition to the correct complex label.

These results reveal consistent, but slightly lower, endorsement of the target labels for the complex expressions in the images. As with the basic expression images, there was also specificity in the perception of the expressions. For example, across all the images in which complex expressions were exhibited, the alternative complex (M = 15.7%, SD = 8.7%) and alternative basic (M = 10.0%, SD = 8.1%) labels were rarely endorsed (see Table 4). The images in which there was the most willingness to endorse an alternative label was for the expression of jealous. On average 25.5% (SD = 4.1) of participants chose the alternative complex label “despondent”. The images with the expressions of affectionate, attracted, betrayed, brokenhearted, contemptuous, desirous, and jealous were most often mistaken for another complex expression. However, lovesick was most often mistaken for one of the alternative basic expressions, either fearful or disgusted.

The full set of images that were endorsed ≥ 50% are available for download for research purposes at Databrary (http://doi.org/10.17910/b7.874).

Discussion

Here, we describe a database of facial emotional expressions posed by formally trained, young adult actors. There are both basic (six) and complex social (nine) expressions, with a particular emphasis on social sexual (e.g., desire, flirtatious) expressions [12]. All of the individual stimuli were validated by a minimum of 50 adult raters and have endorsement scores at or over 50% in a 4AFC paradigm. There are 480 images in the final database: 243 images represent basic expressions and 237 images represent complex expressions.

We provided images that were minimally edited to reflect the way they were rated on MTurk and that met a low threshold of endorsement to provide researchers with flexibility in determining how to work with the stimuli. Importantly, researchers may want to take additional steps with the stimuli to process or control multiple physical characteristics of the images. For example, in our own work, we match images for luminance, use a higher endorsement threshold to down select stimuli for our tasks, and crop the images so that only the head and neck is visible in each image. Researchers may also consider cropping out hair.

There are some limitations to consider regarding these stimuli. First, a central goal of designing these stimuli was to include multiple actors who posed the same expressions to avoid a confound between actor and expression. However, not all actors generated exemplar expressions that surpassed the endorsement threshold. As a result, there is variability in the number of images each actor contributed to the final stimulus set. Second, jealous was a particularly difficult expression to either create or distinguish from other expressions. The stimulus set only contains nine images of the jealous expression (all male) that met the minimum endorsement criterion. Third, the complex expressions in the database are all of the social sexual category because of our own work and hypotheses. It will be essential that similar databases with social self-conscious and other kinds of social complex or cognitive complex expressions be created and validated.

It is important to note that the focus of this study was to validate the external validity of the stimuli as complex emotional expressions. Our goal was to amass ratings from a large, representative sample of adults to help validate the stimuli. As a result, we tested people who self-identified as male, female, and non-binary genders, and who self-reported being 18–82 years of age. Importantly, we did not design this study as an empirical investigation of the effects of these participant factors (e.g., gender, age) on the perception of the expressions. This requires careful balancing of groups for confounding factors and methodological procedures (e.g., screening for psychopathology which can vary by gender). However, these are exactly the kinds of questions we hope researchers will pursue using the CEED stimuli. Specifically, are there effects of gender and/or age of the observer and/or stimulus face on the perception of basic/complex expressions?

By making CEED available to researchers, this will support the study of the developmental, behavioral, and neural basis of facial emotion expression processing. This database will enable researchers to study the perception and recognition of complex social expressions, particularly social sexual expressions, which have previously been underrepresented in the literature. Using the basic and complex stimuli available in CEED, future researchers can develop a better understanding of how perception of basic and complex expressions differ, how sensitivity to certain emotion expressions develops over the lifespan, and how social contexts can impact the perception of emotion expressions. The focus on complex expressions in this database will begin to fill in the gaps in the current literature and allow for nuanced questions about emotion expression to be addressed.

Emotion labels.

This table includes the emotion label answer choices presented to the MTurk raters in the validation task. It lists the four emotion labels presented for each of the 15 types of facial emotion expression. This includes the target label expression, the alternative basic expression(s), and the alternative complex expression(s).

(DOCX)

Click here for additional data file.

Image ratings.

Image ratings for all stimuli included in CEED. Includes information regarding the number and gender of raters, the answer choices provided with the image, and the percentage of raters who chose each answer choice. The images are listed by expression type, actor, and file name.

(XLSX)

Click here for additional data file.

30 Oct 2019

PONE-D-19-25816

The Complex Emotion Expression Database: A validated stimulus set of trained actors.

PLOS ONE

Dear Dr Scherf,

Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process.

We would appreciate receiving your revised manuscript by Dec 14 2019 11:59PM. When you are ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter.

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. For instructions see: http://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols

Please include the following items when submitting your revised manuscript:

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). This letter should be uploaded as separate file and labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. This file should be uploaded as separate file and labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. This file should be uploaded as separate file and labeled 'Manuscript'.

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out.

We look forward to receiving your revised manuscript.

Kind regards,

Zezhi Li, Ph.D., M.D.

Academic Editor

PLOS ONE

Journal Requirements:

1. When submitting your revision, we need you to address these additional requirements. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at http://www.journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and http://www.journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf

2. We note that Figures 1 and 2 includes an image of a [patient / participant / in the study].

As per the PLOS ONE policy (http://journals.plos.org/plosone/s/submission-guidelines#loc-human-subjects-research) on papers that include identifying, or potentially identifying, information, the individual(s) or parent(s)/guardian(s) must be informed of the terms of the PLOS open-access (CC-BY) license and provide specific permission for publication of these details under the terms of this license. Please download the Consent Form for Publication in a PLOS Journal (http://journals.plos.org/plosone/s/file?id=8ce6/plos-consent-form-english.pdf). The signed consent form should not be submitted with the manuscript, but should be securely filed in the individual's case notes. Please amend the methods section and ethics statement of the manuscript to explicitly state that the patient/participant has provided consent for publication: “The individual in this manuscript has given written informed consent (as outlined in PLOS consent form) to publish these case details”.

If you are unable to obtain consent from the subject of the photograph, you will need to remove the figure and any other textual identifying information or case descriptions for this individual.

Additional Editor Comments:

There are some concerns of two reviewers which should be addressed before we can consider its potential publication.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

Reviewer #2: No

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: Comment on PONE-D-19-25816

The paper entitled “The Complex Emotion Expression Database: A validated stimulus set of trained actors” By Margaret S. Benda et al introduced a Complex Emotion Expression Database (CEED), a digital stimulus set of 243 basic and 237 complex emotional facial expressions. Images were validated by around 50 adults in a 4-alternative forced choice task and shown in tables with those have higher than 50% raters endorsement and correct emotion labels.

Overall, the study is well designed. Results are well described. The authors have good discussions about their findings.

However, the following concerns need to be addressed:

All images were performed by young adult actors with ages ranging from 18 – 27 years old, while the raters are from 18 – 82 years old. It is possible that older people are more developed in their cognitive skills thus have different opinions (compared with young raters) towards the concept of the same expression. The author may want to discuss this point.

In the method section, the authors indicated that “To determine the specificity of the endorsement scores, we also computed the frequency with which raters picked the alternative labels.” Please introduce more details about how the frequency the raters picked the alternative labels affects the endorsement scores.

Do genders have effects on the judges of “correct expressions”?

The authors give several alternative labels for the same expression. It is possible that in some expression group the labels have quite similar meanings which make it difficult to judge correctly, therefore the expression has a low endorsement score. The author may want to discuss whether the types of alternative labels given affect the endorsement scores.

Other minor correction:

Please use justify text for the manuscript.

Reviewer #2: I am very happy to have the opportunity to review this article. Overall, I feel that it is not suitable to publish this paper on our magazine of high level. The main shortcoming is the introduction part. The author simply say that the specific mechanism of research on face recognition is unknown. In fact, there are still many positive results on mechanisms such as Neural pathway, biochemical mechanism etal. At the same time, the author does not explain the complex emotions in a more reasonable way. It is better to have a hypothesis instead of just comparing it with other face databases while this article also missed the introduction of the EKMan expression database which is recognized as a comprehensive expression library.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files to be viewed.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email us at figures@plos.org. Please note that Supporting Information files do not need this step.

Submitted filename: review of plosone1029.doc

Click here for additional data file.

13 Dec 2019

Response to Reviewers:

In general, the Reviewers and Editor were positive about the study, rating it as technically sound with rigorous statistical analyses and publicly available data and stimuli. The Editor noted that this study “has merit.” Reviewer 1 indicated that the study “is well designed,” the “results are well described,” and that we provided “good discussions about the findings.”

Reviewer 1’s concerns were focused on requests for clarification and for additional information about how characteristics of the raters may have influenced the results of the validation study and how the endorsement scores were computed. Reviewer 2 requested additional information about the nature of complex emotional expressions. We have addressed each of these concerns below with detailed responses. We have also corrected all typographical errors and formatted the manuscript to be compliant with PLoS One requirements. We think the paper is much improved as a result.

Reviewer 1:

Although the stimuli included individuals ages 18-27 years, the raters included individuals ages 18-82 years. Is it possible that there are age-related effects in the perception of these expressions? The authors may want to discuss this point.

We agree with the Reviewer that this is an interesting possibility and exactly the kind of question we hope researchers will investigate using the CEED stimuli. Unfortunately, because this is a validation study of the stimuli; it was not designed to address this question empirically. For example, to investigate age-related effects on the perception of the stimuli, we would need to design the study ahead of time to ensure that there are equal numbers of male and female participants in the relevant age groups and we would have to clearly define the age groups a priori based on hypotheses. This study was not designed in this way.

However, we think this point reflects a concern on the part of the Reviewer about whether age-related effects on perception of the stimuli could affect the results of the validation study. In other words, if older raters had consistently higher endorsement scores on the stimuli, did more stimuli reach the 50% threshold for inclusion in the database? We agree that this is an important issue to address to the best of our abilities. Therefore, we looked at the mean endorsement scores for older and younger participants (defined by median split on age � 31 years) for several exemplar stimuli in each expression, including the highest-, lowest- and a middle-rated images. Across these images/expressions, there was no clear pattern in which either the younger or older participants consistently rated the images higher/lower. Therefore, we are confident that there is no consistent age-related influence on the endorsement ratings of these expressions that might impact the results of the validation study and the specific set of images that are included in CEED.

Given the importance of this point, we have added a paragraph in the Discussion that addresses these issues and recommends that this work be done in the future using the CEED stimuli. It reads (p. 18-19):

“It is important to note that the focus of this study was to validate the external validity of the stimuli as complex emotional expressions. Our goal was to amass ratings from a large, representative sample of adults to help validate the stimuli. As a result, we tested people who self-identified as male, female, and non-binary genders, and who self-reported being 18-82 years of age. Importantly, we did not design this study as an empirical investigation of the effects of these participant factors (gender, age) on the perception of the expressions. This requires careful balancing of groups for confounding factors and methodological procedures (e.g., screening for psychopathology which can vary by gender). However, these are exactly the kinds of questions we hope researchers will pursue using the CEED stimuli. Specifically, are there effects of gender and/or age of the observer and/or stimulus face on the perception of basic/complex expressions?”

Do genders have effects on the judges of “correct expressions”?

This is also a very interesting possibility that we did not design the validation study to investigate. When we design studies to investigate the potential effects of sex-differences on face processing, we are very careful to screen participants for ongoing symptoms or personal and family history of psychopathology, which can impact face processing and which is differentially represented across sex (see Scherf et al., 2017). Also, we make sure that there were no age differences across male and female participants. Neither of these critical methodological controls was implemented in this validation study. Therefore, although we know that 51% of the raters of the final set of stimuli self-identified as male and 49% self-identified as female, we do not have empirical data about sex differences in the perception of the CEED stimuli. To address the importance of this issue, we also mention the need to investigate the potential role of sex differences on the perception of these stimuli in the Discussion (p.18-19).

Please introduce more details about how the frequency the raters picked the alternative labels affects the endorsement scores. Researchers can determine whether raters confuse the expression more other another complex expression or another basic expression.

Thank you for the suggestion. Recall that the endorsement score was defined as the percent of total raters who endorsed the target label for the expression (i.e., picked the label “angry” for an angry expression).

We have addressed this concern and followed the recommendation in two ways. First, in the Methods section (p. 13), we have now clarified:

“This [the endorsement score] is essentially an accuracy score if one considers the target label the “correct” label. Therefore, the frequency with which participants pick an alternative label reflects the “error” on a given trial.

and

“In addition, we analyzed the error on each item to identify the proportion of raters who chose each of the alternative labels. For example, researchers can determine whether raters are likely to misidentify an expression (e.g., anger) consistently with the label for another expression (e.g., betrayed) or not. Together with the endorsement scores, this error information helps to define the relative specificity of these scores for each stimulus.”

We also highlighted findings from the recommended analysis in the Results section (p.15).

“Also, it is important to note that although the basic expression labels were repeated more frequently as alternative labels (because there are fewer to choose from within valence), this did not bias raters’ selection of alternative labels. Supplementary Table 2 shows that for all basic expressions, the most common alternative label picked was a complex (not basic) expression label. Similarly, for complex expressions, the most common alternative label picked was also a complex label, even though there were two basic label options.

The authors may want to discuss whether the types of alternative labels given affect the endorsement scores.

To address this concern, we have provided more information about how we generated the alternative labels, particularly for the complex expressions. This is described in the Methods section on p. 12. It reads:

“The labels describing complex expressions were selected from the labels used in the Revised Reading the Mind in the Eyes Test (RMET) [40]. The RMET is a commonly used measure of emotion expression perception that includes complex expressions and presents participants with a similar 4-alternative forced choice paradigm.”

We purposefully avoided picking words that were semantically similar (e.g., fearful, terrified) to the target label.

Reviewer 2:

The main shortcoming is the introduction part. The author simply say that the specific mechanism of research on face recognition is unknown. In fact, there are still many positive results on mechanisms such as Neural pathway, biochemical mechanism etal

We think this concern reflects a misreading of our paper. Our reference to “mechanism” in paragraph 2 of the Introduction explains that the vast majority of existing work in the current literature is

“investigating the developmental, behavioral, and neural mechanisms supporting the perception and recognition of facial expressions is focused on basic expressions.”

This is important because the mechanisms could be different, similar, or overlapping for perceiving and recognizing complex expressions. To make this point explicitly clear, we added the following sentence to the end of that paragraph (p.3):

“The developmental, behavioral, and neural mechanisms supporting the perception and recognition of complex expressions could be similar, different or overlapping. There is some existing work suggesting that the developmental and behavioral mechanisms may be different [see 12-13].”

The author does not explain the complex emotions in a more reasonable way. It is better to have a hypothesis instead of just comparing it with other face databases.

We would like to point out that we provided a clear definition for complex expressions in the Introduction of the manuscript on p. 3. It explains that complex expressions “provide signals about emotions related to more nuanced social behavior and inner thoughts” and comes from Baron-Cohen and colleagues (1997). We cited Garcia & Scherf (2015) because we review this paper together with additional evidence supporting the distinction between basic and complex emotions/expressions. Also, there is a section of the Introduction devoted to explaining complex social expressions under the heading – “Categories of complex emotional expressions” (p. 4). In this section, we describe categories of social complex expressions and the capacity (or lack thereof) of existing databases to be used to address scientific questions about the underlying developmental, behavioral and neural mechanisms supporting the processing of complex expressions. This is critical for demonstrating the critical need for a database of stimuli like CEED.

It is also important to note that this is a validation study to assess the external validity of the stimuli, not a study designed to investigate hypothesis-driven questions about the perception of these stimuli. We hope researchers will use the stimuli for exactly that purpose.

This article also missed the introduction of the EKMan expression database which is recognized as a comprehensive expression library.

Again, we think this is a misreading of our paper. We reference the Ekman & Friesen (1971) paper in which the original face stimuli were presented in the second paragraph of the Introduction (p. 3). It is reference number 10. Importantly, this database only includes basic expressions (happy, angry, disgusted, fearful, surprised, sad). This is why this paper is not listed in Table 1, which only lists databases that include complex expressions.

Submitted filename: CEED_Response_to_Reviewers.docx

Click here for additional data file.

13 Jan 2020

The Complex Emotion Expression Database: A validated stimulus set of trained actors.

PONE-D-19-25816R1

Dear Dr. Scherf,

We are pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it complies with all outstanding technical requirements.

Within one week, you will receive an e-mail containing information on the amendments required prior to publication. When all required modifications have been addressed, you will receive a formal acceptance letter and your manuscript will proceed to our production department and be scheduled for publication.

Shortly after the formal acceptance letter is sent, an invoice for payment will follow. To ensure an efficient production and billing process, please log into Editorial Manager at https://www.editorialmanager.com/pone/, click the "Update My Information" link at the top of the page, and update your user information. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, you must inform our press team as soon as possible and no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

With kind regards,

Zezhi Li, Ph.D., M.D.

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: No

23 Jan 2020

PONE-D-19-25816R1

The Complex Emotion Expression Database: A validated stimulus set of trained actors.

Dear Dr. Scherf:

I am pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximize its impact. If they will be preparing press materials for this manuscript, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

For any other questions or concerns, please email plosone@plos.org.

Thank you for submitting your work to PLOS ONE.

With kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Zezhi Li

Academic Editor

PLOS ONE