The spatio-temporal features of perceived-as-genuine and deliberate expressions.

Reading the genuineness of facial expressions is important for increasing the credibility of information conveyed by faces. However, it remains unclear which spatio-temporal characteristics of facial movements serve as critical cues to the perceived genuineness of facial expressions. This study focused on observable spatio-temporal differences between perceived-as-genuine and deliberate expressions of happiness and anger expressions. In this experiment, 89 Japanese participants were asked to judge the perceived genuineness of faces in videos showing happiness or anger expressions. To identify diagnostic facial cues to the perceived genuineness of the facial expressions, we analyzed a total of 128 face videos using an automated facial action detection system; thereby, moment-to-moment activations in facial action units were annotated, and nonnegative matrix factorization extracted sparse and meaningful components from all action units data. The results showed that genuineness judgments reduced when more spatial patterns were observed in facial expressions. As for the temporal features, the perceived-as-deliberate expressions of happiness generally had faster onsets to the peak than the perceived-as-genuine expressions of happiness. Moreover, opening the mouth negatively contributed to the perceived-as-genuine expressions, irrespective of the type of facial expressions. These findings provide the first evidence for dynamic facial cues to the perceived genuineness of happiness and anger expressions.

Introduction

People with perceived-as-genuine smiles are often judged as being more attractive, friendly, and trustworthy than those who show perceived-as-deliberate smiles, thereby eliciting cooperative behaviors from decoders [1]. In contrast, perceived-as-genuine angry expressions read from a sport’s team coach may cause players to cower while playing their sport [2]. Given the endogenous nature of perceived genuineness posited to increase the trustworthiness of the expresser by communicating the need to embark upon and ensure successful social interaction [3], perceived-as-genuine expressions can be expected to have more significant impacts on decoders’ behavior when compared with perceived-as-deliberate expressions. Indeed, Krumhuber et al. [4] revealed that perceived-as-genuine smiling interviewees were more likely selected for the simulated job. Recent studies also have demonstrated that perceived-as-genuine expressions, more than perceived-as-deliberate ones, make decoders behave pro-socially in several experimental settings [3-6].

It remains unclear, however, what facial morphological features and spatio-temporal dynamics drive the perceived genuineness of facial expressions. For the morphological aspects of genuine facial expressions, the Duchenne smile has been described as one of the most famous representatives of the genuine expression [7]. The Duchenne smile is defined as a smile that involves the activation of the orbicularis oculi muscle (raising the cheek), and it is known that the genuineness of positive emotions perceived from encoders depends on whether the cheek is raised [1]. Originally, the Duchenne smile was associated with signs of positive emotions, such as enjoyment [7-10]. However, a recent study has suggested that raising the cheek can be regarded as an artifact of smile intensity rather than an indicator of positive emotion [11]. As for the temporal aspects, genuine smiles—more than deliberate ones—had longer durations between the onsets and offsets of lip corner movements [12-15]. Perusquía-Hernández et al. [16] also reported that an electromyography-based automatic detection machine trained with the temporal dynamics of smiles was able to discriminate genuine smiles from deliberate ones. More recently, Sowden et al. [17] demonstrated, using facial landmarks, that speed of facial movements differentiates deliberate expressions of anger, happiness and sadness. On the other hand, Ambadar et al. [18] clearly acknowledged the difficulty of determining whether encoders’ intended meanings agreed with those perceived by decoders. Although facial expression clues from the encoders’ perspective influence perceptions and judgments of smile genuineness [4, 6, 19], decoders’ perceived meanings and encoders’ genuine expressions must be investigated. Considering that facial expression information depends on the decoder’s interpretation, evidence that encompasses both perspectives would result in a deeper understanding of facial expressions.

Beyond the aspect of the encoder, some studies have investigated facial expressions from decoders’ interpretations [20]. For example, using randomly generated facial movements in avatars and their decoders’ categorizations based on specific emotions, functions, and affect grids, Jack and their colleagues found that face movements matched these categories [21-23]. Although this data-driven approach has provided outstanding findings on the spatio-temporal features of facial expressions that correspond to the decoders’ interpretation, the practical constraint on the kinetic potential of facial expressions is not guaranteed from the ecological validity viewpoint as decoders have observed facial avatars rather than real human faces. Further, the particular spatio-temporal features most important in the human perception of what is genuine vs deliberate remain an open issue. To further understand the spatio-temporal features of facial expressions, it would be desirable to investigate actual human facial expression movement instead of avatars and to compensate for them.

The current study aimed to clarify the spatio-temporal features of perceived-as-genuine facial expressions by having participants judge whether real human faces show genuine or deliberate emotions on the basis of their facial movements. Dawel et al. provide genuine/false norms for facial expressions, but their analysis mainly relies on visual inspection of facial photographs without a quantitative analysis of the spatio-temporal features of facial expressions [24]. Ambadar et al. [18] also suggest that perceived-as-amused smiles consist of enhanced cheek raising, an open mouth with a larger amplitude, and a longer duration than perceived-as-polite smiles. However, there are two methodological limitations in the study. First, the number of coded facial movements is limited. Moreover, the number of video frames required to record spontaneous facial expressions differ, which makes it difficult to quantitatively compare between perceived-as-amused and perceived-as-polite smiles. To overcome the methodological problems, the current study developed perceived-as-genuine/deliberate expressions and examined their spatio-temporal features using deliberate expressions’ facial databases, in which the number of frames and position of peak are controlled. Furthermore, we tested anger expressions as well as happy expressions, whereas many other scholars have only studied happy expressions. It is important to investigate perceived-as-genuine anger because the decoders’ interpretation of angry facial expressions depends on the genuine vs. deliberate axis as much as happy ones do [25].

More concretely, the participants in this study judged the genuineness of a set of dynamic facial databases of happiness and anger. Then, this study explored which spatial pattern related to the decoders’ judgment of genuineness, using a mixed model that explicitly modeled encoder and decoder effects. After that, we identified the spatio-temporal features of the perceived-as-genuine and perceived-as-deliberate facial expressions of happiness and anger, using a state-space model with change point detection of spatial component changes over time [26, 27].

We anticipated that the spatial patterns of both expressions would correspond to the prototypical expressions predicted by basic emotion theory (BET) [28]. Krumhuber et al. [29] found that deliberate expressions were more prototypical in their facial patterns than spontaneous ones. Therefore, we expected that these prototypical spatial patterns would decrease the decoders’ judgment of genuineness and be enhanced in perceived-as-deliberate expressions more than in perceived-as-genuine expressions. The prototype of happiness is a smile with a contraction of the orbicularis oculi muscle, while the prototype of anger is facial movements composed of lowering the eyebrows, widening the eyes, and tightening the lower eyelids. It should be noted that this study did not aim to evaluate the validity of facial expressions based on the BET [30, 31]. Considering the previous findings for the temporal patterns from encoders [12-15], we anticipated that the onset would be faster with perceived-as-deliberate expressions than with perceived-as-genuine expressions.

Methods

Participants

A total of 89 crowdsourcing workers (64 women and 25 men: age range = 19–73, Mean = 37.92, SD = 10.79) agreed to participate in a survey via Crowdworks (CW: www.crowdworks.jp), and all participants were Japanese. The validation of CW participants has already been confirmed by Majima et al. [32] and is aligned with that of the normal participants of behavioral experiments. Informed consent on the CW platform was obtained from each participant before the investigation in line with a protocol approved by the Ethical Committee of the Graduate School of Education, Hiroshima University (2019086), and the Institutional Review Board of Waseda University (2015–033). This study was conducted in accordance with the ethical guidelines of our institute and the Declaration of Helsinki. After completing the experimental task, the participants received 900 JPY for completing a 60-min survey.

Stimuli

This study used prerecorded video clips of facial expressions from 20 Japanese models (50% women: age range = 21–33, mean = 26.60, SD = 3.22). This dynamical facial database was developed by another research project. The models were asked to show facial expressions according to six emotions (anger, happiness, disgust, fear, sadness, and surprise) under four emotional scenarios and to show a neutral expression four times. The models were instructed to maintain a neutral expression for the initial 4 seconds and then show an intended emotion on their faces for 5 seconds in a way they thought natural. To aid the models in producing their expressions according to the time course, the timing of initiating expressions was indicated by a pure tone (1000 Hz) produced from a speaker system, followed by sound presentation every second. This instruction aimed to show the models how to deliberately produce their facial expressions within a certain time range to make it easier to compare between expressions at the expense of the natural time course of facial expressions. All video sequences had 1920 x 1440 pixel resolutions at 30 frames per second and were targeted, ranging from −2000 ms to +2000 ms from the onset of facial movements (start of a pure tone), resulting in 121 frames (4 seconds). The current study extracted only three types of emotions (i.e., anger, happiness, and neutral), of which anger and happiness of the same two men and women were excluded, due to time constraints and human resources involved in the viewed expressions. Consequently, the current study used 16 (models) x 2 (emotion: anger, happiness) x 4 (scenarios) plus neutral expressions by 20 (models: 148 total clips). Example stories of anger and happiness are the following: “when you are blamed even though you are not at fault at all (angry1),” “when someone insults your family (angry2),” “when you enjoy conversation with your friends (happy1),” and “when someone praises you (happy2).”

Procedure

This study used the Gorilla Experiment Builder (www.gorilla.sc) to create and host our experiment [33]. Data were collected between November 29, 2019, and December 27, 2019. All participants were asked to provide consent via a check-box if they wished to participate. Thus, written type of consent was informed and obtained. This form of consent was approved by the Ethical Committee of the Graduate School of Education, Hiroshima University. This was the only form of consent that was given. On the experimental platform, the participants provided some basic demographic information (age and sex). After this, they were given careful instructions about the concept of genuine and deliberate facial expressions and their requirements as participants, followed by Namba et al. [34]. The following instruction was given in Japanese: “People sometimes express genuine facial expressions caused by actual emotional experiences, while some people can express deliberate facial expressions of emotion by intentional manipulation. In this study, we aim to understand whether people have the ability to detect whether or not the person depicted is feeling each emotion.” Unknown to the participants, all expressions were deliberate. Next, all the participants performed practice trials with two facial stimuli not used in the main trials (two intended smiles expressed by the experimenter). When the participants completed the practice trials, the platform confirmed that the participants understood the task. If the participants responded with no questions, the main trials began. However, if there were issues understanding the task, the participants were reminded of the instructions and asked to redo the practice trial. The main task program presented expressions from a pool of 148 dynamic facial stimuli. We asked participants to judge whether the target person expressed genuine or deliberate expressions. The order of facial stimuli was randomized. All clips were played once, and the inter-stimulus interval was exactly 300 ms.

Following the main task, the participants filled out the Japanese version of four questionnaires related to social cognition: the Social Interaction Anxiety Scale [35, 36], the Social Phobia Scale [35, 36], the Emotional Contagion Scale [37, 38], and the Interpersonal Reactivity Index [39]. These metrics were measured for another relevant research project [40] on emotional perception, and thus we did not report the results using these questionnaires.

Statistical analysis

To the happy (N = 64) and angry (N = 64) facial stimuli, we extracted frame-level action unit (AU) intensities on a 5-point scale with an automatic AU detection system (Openface [41, 42]). The Facial Action Coding System considers AUs as having the ability to describe all facial movements anatomically [28]. While OpenFace does not guarantee the same performance that manual facial coding does, there was sufficient biserial correlation (r = .80) between OpenFace and expert FACS coders’ performances to static frontal facial images of Japanese persons [43]. OpenFace can detect 18 AUs: 1 (inner brow raiser), 2 (outer brow raiser), 4 (brow lowerer), 5 (upper lid raiser), 6 (cheek raiser), 7 (lid tightener), 9 (nose wrinkler), 10 (upper lip raiser), 12 (lip corner puller), 14 (dimpler), 15 (lip corner depressor), 17 (chin raiser), 20 (lip stretcher), 23 (lip tightener), 25 (lips parts), 26 (jaw drop), 28 (lip suck), and 45 (blink).

To reduce the dimensionality and extract the low-dimensional features, a nonnegative matrix factorization was applied to the time-series data of the AUs [44-46]. This approach helps obtain interpretable features in a low-dimensional space [44]. Indeed, the nonnegative matrix factorization [47] is the space-by-time manifold algorithm and is suitable for identifying the dynamic facial patterns that extract spatial (AU combination) patterns with reduced dimensions and time-series changes [48, 49]. Chiovetto et al [50] also permitted very low-dimensional parametrization of the associated facial expression with emotion, using a similar approach. The factorization rank was determined by the cophenetic coefficients [51].

To clarify the relationships between identified NMF patterns and decoders’ dichotomous judgments of them as genuine or deliberate, a generalized linear mixed model was conducted to control for the differences between each encoder and decoder. In addition, we adopted a Bayesian approach to evaluate uncertainty as probability distributions. The models in this study are described as follows:

All predictors were standardized to improve the interpretation of the coefficients. All priors were kept at the default settings for the brm function [52]. If the 95% credible interval of the parameters does not include zero, a significant effect could be inferred to have been identified.

Based on the decoders’ dichotomous judgments of the presented expression as genuine or deliberate, we divided facial expressions into the following three types: the relatively perceived-as-genuine, the ambiguous, and the relatively perceived-as-deliberate facial expressions. Of the happy/angry facial stimuli, we extracted the +0.8/+1.0 SD adjudged genuine, as well as the −0.8/−1.0 SD stimuli adjudged deliberate (Fig 1). Finally, the number of target facial expressions was 128 (16—eleven women and five men—perceived-as-genuine happiness; 36—eighteen women and men—ambiguous happiness; 12—three women and nine men—perceived-as-deliberate happiness; 13—ten women and three men—perceived-as-genuine anger; 37—nineteen women and eighteen men—ambiguous anger; 14—three women and eleven men—perceived-as-deliberate anger). Taking each frame (121) in each video resulted in 15,488 data points (121 frames x 128 expressions). These expressions were employed to systematically generate facial expressions considered perceived-as-genuine/ambiguous/deliberate expressions and not the same as participants’ to estimate population indices for effect sizes. Consequently, power analyses were not available. The N of 64 for each emotion was chosen as more than the usual number of expressers employed in the research using the actor’s facial expressions, which was likely to produce stable means and allow for conducting multivariate statistical analyses [53]. Moreover, this sample size is expected to emphasize the more distinctive descriptions of each perceived-as expression.

Fig 1

The histogram of Yes responses for the genuineness judgment of happiness (upper part) and anger (lower part).

For the temporal features, we applied a state-space model with the change point detection to spatial component changes over time [26, 27]. The model can be described as follows: where Y are the observable matrices of the spatial component matrix, and t means the frame or time. μ is the spatial component matrix common to three expressions. δ1 / δ2 can be considered the magnitude of difference between the perceived-as-genuine/deliberate/ambiguous expressions. A prior distribution without any specification is a uniform distribution. The code is available on Open Science Framework (OSF: https://osf.io/e7pdt). If the δ terms are greater than zero (i.e., positive value), this means that the spatial component of perceived-as-genuine/deliberate is relatively large, and if it is smaller than zero (i.e., negative value), this means that the spatial component of perceived-as-genuine/deliberate expressions is relatively smaller than that of ambiguous expressions. We calculated the 99% credible interval of the δ as to whether the intervals fall to zero could be considered as the testing for δ.

To develop the spatio-temporal patterns from AU data, we used the “NMF” packages [54] in R to implement the calculation. As for the generalized linear mixed model, all iterations were set to 3,000 and burn-in samples were set to 1,000, with the number of chains set to four using the “brms” package [52]. For a state-space model, we used the “cmdstanr” package [55] and set all iterations to 15,000, as well as burn-in samples to 5000. The value of R-hat for all parameters equaled about 1.0, indicating convergence across the four chains [56].

Results

Happiness

Fig 2 shows the spatial components from all facial expressions of happiness. Visually inspecting the relative contribution of each AU to the independent components, we interpreted Component 1 as opening the mouth (AU25, 26). The results of Component 2 indicated smiling (AU12) with eye constriction (AU6, 7) and opening the mouth (AU25), while those of Component 3 suggested that raising the chin (i.e., AU17) was a main contributor. Although Component 2 also included upper lip raising (AU10) and dimpling (AU14), these AUs can be interpreted as the confusion of AU12 in the automated action coding detection system [46, 57].

Fig 2

Heatmap of each component’s loadings for facial expressions of happiness (upper part) and visual representations (lower part). Value colors represent each facial movement’s contribution to component scores.

To clarify the relationships between identified NMF patterns and decoders’ dichotomous judgments of them as genuine or deliberate, a generalized linear mixed model with random intercepts was built and tested to control for the differences between each encoder and decoder. Table 1 depicts the coefficients for each factor of NMF predicting genuineness judgment. Notably, Component 1 (opening the mouth) and Component 2 (smiling with eye contraction) were found to predict genuineness judgment (β1 = −0.78, 95% Credible Intervals [−1.07, −0.50]; β2 = −0.46, 95% CI [−0.75, −0.18]), but Component 3 (raising the chin) did not because of the 95% CI that included 0 (β3 = 0.10, 95% CI [−0.17, 0.37]).

Table 1

Results of the generalized linear mixed model for the relationships between identified NMF patterns and decoders’ dichotomous judgments of genuineness.

	Happiness	Anger
Random effects	Variance [95%CI]
Decoders (intercept)	0.89 [0.75, 1.06]	1.21 [1.01, 1.44]
Encoders (intercept)	1.00 [0.82, 1.22]	0.88 [0.72, 1.08]
Fixed effects	EAP [95%CI]
Component 1	-0.78 [-1.07, -0.50]	-0.62 [-0.85, -0.38]
Component 2	-0.46 [-0.75, -0.18]	-0.23 [-0.48, 0.01]
Component 3	0.10 [-0.17, 0.37]	-0.39 [-0.64, -0.15]

To differentiate perceived-as-genuine and perceived-as-deliberate facial expressions of happiness, Fig 3 shows the quantitative indices of the time-series patterns for the magnitude of difference between the perceived-as-genuine, ambiguous and perceived-as-deliberate expressions of happiness. S1 Table represents the 99% credible intervals and probability of directions [58, 59] at 500 ms intervals. Visual inspection of Component 1 (opening the mouth) revealed that the perceived-as-deliberate expressions showed a larger mouth opening, while the perceived-as-genuine expressions remained deactivated when compared with ambiguous expressions. As for Component 2 (smiling with eye contraction), the perceived-as-deliberate expressions produced more rapid facial changes than the perceived-as-genuine expressions. At the middle row in the right-hand-side column of Fig 3, the difference parameter (i.e., δ1 - δ2) clearly indicated that the perceived-as-deliberate expressions reached their peaks earlier than the perceived-as-genuine expressions did. Unexpectedly, ambiguous expressions showed a stronger smiling component as offset areas (after peak: 501–2000 ms) than the other two expressions did. Component 3 (raising the chin) can be interpreted as a byproduct of Component 1 because it corresponds to raising the chin, which also means the movement of closing the mouth.

Fig 3

Time-series patterns for the magnitude of difference between the perceived-as-genuine and perceived-as-deliberate expressions of happiness.

The y-axis represents the extent of the “δ” parameters for each component. Solid lines indicate the expected a posteriori. Positive values refer to a relatively large spatial component of (left: perceived-as-genuine, center: deliberate, right: genuine), while negative values indicate a relatively large spatial component of (left and center: perceived-as-ambiguous, right: deliberate). The ribbons represent 99% credible intervals.

Anger

Fig 4 shows the spatial components from all facial expressions of anger. A visual inspection of Fig 4 shows that Component 1 was contributed to by tightening the eyelids (AU7), opening the mouth (AU25), lowering the brows (AU4), and slightly raising the upper lip (AU10). Component 2 was related to opening the mouth (AU25, 26) and lowering the brows (AU4). The results of Component 3 correspond to raising the chin (AU17).

Fig 4

Heatmap of each component’s loadings for facial expressions of anger (upper part) and visual representations (lower part). Value colors represent each facial movement’s contribution to component scores.

A generalized linear mixed model with random intercepts showed the coefficients for each factor of NMF predicting genuineness judgment (Table 1). All Components were found to predict genuineness judgment (β1 = −0.62, 95% Credible Intervals [−0.86, −0.48]; β3 = −0.39, 95% CI [−0.64, −0.15]), but 95% CI on only Component 2 (opening the mouth) included zero slightly (β2 = −0.23, 95% credible intervals [−0.48, 0.01]).

To differentiate the perceived-as-genuine and perceived-as-deliberate facial expressions of anger, Fig 5 indicates the quantitative indices of the time-series patterns for the magnitude of difference between the perceived-as-genuine, ambiguous, and perceived-as-deliberate expressions of anger. S2 Table represents 99% credible intervals and probability of directions at 500 ms intervals. The perceived-as-deliberate expressions contributed to Component 1, which can be regarded as multiple facial movements more so than the ambiguous and perceived-as-genuine expressions. Moreover, the perceived-as-genuine expressions showed less Component 1 (multiple frown) than the ambiguous expressions did. Component 2 (opening the mouth) had a larger peak in the perceived-as-deliberate and ambiguous expression than it did in the perceived-as-genuine expression. Component 3 (raising the chin) can be interpreted as the byproduct of Component 2 because it corresponds to raising the chin, which also indicates the movement of closing the mouth. As shown in S2 Table, there were differences between perceived-as-genuine vs. ambiguous but not perceived-as-deliberate vs. ambiguous in Component 2 after peak (0–2000 ms).

Fig 5

Time-series patterns for the magnitude of difference between the perceived-as-genuine and perceived-as-deliberate expressions of anger.

The y-axis represents the extent of the “δ” parameter for each component. The solid lines indicate the expected a posteriori. Positive values refer to a relatively large spatial component of (left: perceived-as-genuine, center: deliberate, right: genuine), while negative values indicate a relatively large spatial component of (left and center: perceived-as-ambiguous, right: deliberate). The ribbons represent 99% credible intervals.

Discussion

The current study explored the relationships between the spatial patterns of facial expressions and decoders’ dichotomous judgments of them as genuine and clarified the spatio-temporal features of perceived-as-genuine and perceived-as-deliberate facial expressions. We anticipated that perceived-as-deliberate expressions would show spatial patterns typical of facial expressions and more rapid movements than perceived-as-genuine expressions. The results produced four key findings for the spatio-temporal features of perceived-as-genuine/deliberate expressions of happiness and anger. First, some prototypical facial movements were observed for both emotions. For the happiness expression, the prototypical spatial pattern (Component 2: AU6/7 = the movement of orbicularis oculi, AU12 = the movement of the zygomatic major muscle) was observed in both the perceived-as genuine and deliberate expressions. As for the anger expression, lowering the eyebrows and opening the mouth (Component 2: AU4 = corrugator muscle, AU25 = orbicularis oris) were seen in both the perceived-as genuine and deliberate expressions, while the perceived-as-deliberate expression of anger produced several additional facial movements, including prototypical patterns (Component 1: AU4, AU7, AU25). Second, genuineness judgments were reduced when more spatial patterns were observed in facial expressions. More concretely, anger expressions included more multiple frowning (Component 1), opening the mouth (Component 2), and raising the chin (Component 3) and were perceived-as-deliberate, while happiness expressions included more opening the mouth (Component 1) and smiling with eye contraction (Component 2) and were perceived-as-deliberate. Third, the main component of happiness (Component 1) revealed that the perceived-as-deliberate expressions reached their peaks earlier than the perceived-as-genuine expressions. Finally, the movement of opening the mouth in both emotions contributed largely to decoders’ dichotomous judgments of them as deliberate and the perceived-as-deliberate expressions, and the component on AU17 can be considered a byproduct of this. However, the results for opening the mouth were slightly different between happiness and anger, and in anger, the difference was remarkable with the perceived-as-genuine expressions, but the difference between ambiguous and perceived-as-deliberate ones was small. Regarding happiness, the perceived-as-genuine expressions had a small mouth opening, and the perceived-as-deliberate expressions had a large mouth opening.

Importantly, the spatial patterns inherent to prototypicality vary between emotions. As can be seen from Component 2 in Fig 3, the smiles of the perceived-as-genuine and deliberate expressions were similar in their intensity at offset (i.e., at 500–2000 ms after peak), although that of the perceived-as-deliberate expression had relatively abrupt onsets. The smile-related component in both expressions was similar, at least with respect to the final frame, and the difference in genuine/deliberate judgments might be attributable to their temporal features. The result that this spatial pattern influenced the judgment of genuineness (Table 1) also supported the contention that this temporal information is important for perceived-as-deliberate expressions. On the other hand, for anger, lid tightening (AU7), which is a part of the prototypical expressions [28] and mainly contributed to Component 1, showed significant differences between the perceived-as-genuine and deliberate expressions (Table 1 and Fig 5). The results indicate that the perceived-as-deliberate expressions consist of multiple facial actions. Fig 5 confirms that the relationship increases linearly as the degree of perceived-as-deliberate increases. By placing an ambiguous expression as an intermediate term, the current study increased the generalizability of the results. This view, that perceived-as-genuine expressions have fewer multiple frowns, is consistent with recent findings showing that deliberate anger expressions contained various facial movements more than genuine anger expressions in Asian populations [60]. The results raise the possibility that we adapt ourselves to show genuine anger expressions with fewer movements through our experiences, which might affect the judgments in the current study as well.

For the temporal aspects, as shown by Component 2 of the happiness expressions (i.e., smile-related movements shown in Fig 3), the perceived-as-deliberate expressions contained more rapid onsets than the perceived-as-genuine expressions. This result is consistent with previous findings on decoder-based facial cues [18, 61], and it can be concluded that the temporal change of perceived-as-genuine expressions should be slow when compared to the perceived-as-deliberate ones. The indication of Sowden et al [17] that the speed of mouth-widening actions helps differentiate between happy and other emotional expressions for deliberate expressions is consistent with previous findings regarding the encoder aspects. As Fig 5 shows, with regard to anger, there were more rapid and intense onsets in Components 1 and 2 relative to the perceived-as-genuine expressions. The greater the speed the greater the perceived intensity of anger expressions [17], but rapid speeds are not always understood to be natural as found in recent android research [62]. In line with the accumulated evidence, many scholars have already reported that the temporal aspects of facial expressions are important [63-66]. Nevertheless, future studies should bear in mind that the credibility of messages on facial expressions may differ depending on the speed of their expressions.

More interestingly, the movement of opening the mouth in both emotions contributes strongly to the perceived-as-deliberate expressions. Indeed, Namba et al. [67] found a sequence emphasizing the movement to open the mouth in deliberate smiles and Sowden et al. [17] indicated that the high speed of mouth opening was important for posed expressions of happiness. The results provide the first evidence that exaggerated facial expressions, including opening the mouth, are judged to be deliberate and that this can be extended to anger as well as happiness. Especially in perceived-as-genuine (not deliberate) anger, the degree to which the mouth opens becomes smaller. However, Ambadar et al. [18] indicate the opposite results that perceived-as-amused smiles include opening the mouth more often than perceived-as-polite smiles. One possible explanation for this discrepancy is provided by the nature of the target facial database. Ambadar et al. [18] used the smiles that were not performed in response to a request, whereas the current study applied all facial expressions performed under emotional stories with express intentions. In other words, the former’s spontaneous smile with high intensity differs from the latter’s emphasized deliberate smile in that the cause to express and the uncontrolled duration of the expression may influence the interpretation of the intensity of the mouth opening. An alternative explanation is based on cultural differences. Since the target population of the current study was East Asians, who are prone to high context communications [68], Fang et al. [60] also reported that facial expressions are less distinct in Eastern people than in Western people. Jack et al. [69] support this because they revealed that Westerners showed their mental representations of basic emotions with more distinct facial movements when compared to Easterners. The perceived-as-genuine expressions may have been less intense and more ambiguous in terms of opening the mouth, with a context preferentially processed.

The finding for the spatio-temporal features of perceived-as-genuine and deliberate expressions might contribute to a pragmatic understanding of our emotional communication. Many researchers emphasize actual usage for facial expressions of emotion [70-72], but this remains insufficient for how it is actually expressed in daily life. Given that perceived-as-genuine facial expressions sometimes prompted the decoder to behave to the encoders’ advantage [3-6], the spatio-temporal features of perceived-as expressions should induce important suggestion for future work. For example, in android research, this finding, that lower degrees of opening the mouth and prototypical components enhances genuineness, may contribute to the development of more elaborate “emotional” robots, which can be considered perceived-as-genuine. We will need to continue our efforts to acknowledge and describe the complexity of our emotional communication.

Notably, unexpected gender differences were observed in perceived-as-genuine expressions, that is, more female faces were included in perceived-as-genuine expressions, while more male faces were included in perceived-as-deliberate expressions. This might be partly attributed to the higher perceived emotionality, honesty, and trustworthiness often associated with female-appearing facial features [73, 74], which leads to the perceptual bias that female actors show genuine expressions more frequently than male ones. The current study also included more female than male perceivers, which suggests that the gender imbalance in the pool was due to the random collection of CW data. However, as Spies and Sevincer [75] argued, women tend to be more accurate in distinguishing between authentic and nonauthentic smiles, which is consistent with the study’s purpose that is to examine perceived-as-genuine facial expressions compensating for encoders’ genuine expressions.

While the current study showed the spatio-temporal features of perceived-as-genuine and deliberate expressions, there are limitations to be noted here. First, all facial expressions were essentially deliberate by following emotional stories. If genuine expressions have specific associated movements (e.g., [45]), the current facial database cannot be used to identify them. Therefore, future studies would benefit from accumulating empirical findings from human/avatar facial expressions and encoder/decoder perspectives. While the current study used all deliberate human expressions at the expense of ecological validity, this methodology has an advantage in controlling the overall duration and the position of the peak. Previous studies point out that there may be multiple peaks in spontaneous facial reactions [49, 76], and thus, future research will need to take into account such complexity that cannot be investigated in deliberate expressions. Further, 2000 ms before and after the peak of expression were arbitrarily extracted in this study. It has been reported that offset is important for decoders [77]. It is important to consider including complete ranging in offset as opposed to onset when using the other deliberate expression database. Second, the results of this study are only based on Japanese samples. Rychlowska et al. [78] have argued that historical heterogeneity is associated with norms favoring greater emotional expressivity. Niedenthal et al. [79] suggest that historically heterogeneous societies promote expressivity and clarity in emotional expressions. Given that Japan has populations of historically homogeneous societies that share common values and rely on more indirect and ambiguous communication depending on contextual information [80], the finding of the current study can be culture specific. It should also be noted that the experiments could not be controlled well as they were conducted online and several studies have suggested that crowd worker data sometimes do not achieve reliable quality [81]. Therefore, it will be necessary to consider such cross-cultural perspectives in future studies that use laboratory experiments or more online experiments that include attention-check questions. Third, forcing yes-or-no responses from decoders throws away valuable information about the degree of perceived genuineness [82]. Although using the extreme group analysis that the current study applied (i.e., the most perceived-as expressions) has been justified by a simulation study [82], it would be desirable to use a rating scale for authenticity instead of a yes-or-no response because the rating scale’s perceived genuineness of different stimuli is expected to provide much more information [24].

Finally, the current automated evaluation system of the AU can provide several AU intensities at a frame-by-frame level. This is an advantage of using the automated AU detection system; however, it is not perfect despite recent developments in machine learning and artificial intelligence techniques in the area of affective computing [83]. Indeed, for Component 3, the differences between the perceived-as-genuine/deliberate and ambiguous expressions were often observed before the peak frame (Figs 3 and 5). This may reflect noise that is a fit to the individual’s face morphology rather than to facial expressions of emotion. It should be noted that the assessment of facial movements is largely dependent on the target stimuli and their nature [84], but the state-of-the-art AU detection system comparisons provided average F1 scores of .56–.59 [85]. Perusquia-Hernández et al. [46] also indicate the existence of entanglement between upper lip raising (AU10) and lip corner pulling (AU12). Replication studies with a more sophisticated facial movement detection system are awaited.

To summarize, the current study revealed the spatio-temporal features of the perceived-as-genuine and deliberate facial expressions of happiness and anger. In the case of the happiness expression, the smile-related spatial pattern occurred in both perceived-as expressions. For the anger expression, lowering the eyebrows and opening the mouth were seen in both expressions, but the perceived-as-deliberate expression produced multiple facial movements, including squeezing the eyes. In addition, the perceived-as-deliberate expressions had a faster onset to the peak than the perceived-as-genuine expressions. Less movement of opening the mouth in both emotions contributes strongly to the perceived-as-genuine expressions. Identifying the spatio-temporal features of the perceived-as-genuine expressions can contribute to building facial databases that can evoke decoders’ reactions based on the credibility of the nonverbal message. Moreover, it may enrich the affective computing areas by applying to humanoid robots that purport to express human-like displays.

Results for the magnitude of difference between the perceived-as-genuine and perceived-as-posed expressions of happiness compared to ambiguous expressions.

(XLSX)

Click here for additional data file.

Results for the magnitude of difference between the perceived-as-genuine and perceived-as-posed expressions of anger compared to ambiguous expressions.

(XLSX)

Click here for additional data file.

13 Apr 2022

A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.

A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.

An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'.

If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter.

If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols.

We look forward to receiving your revised manuscript.

Kind regards,

Steven R. Livingstone

Academic Editor

PLOS ONE

Journal requirements:

When submitting your revision, we need you to address these additional requirements.

1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at

https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and

https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf.

2. Please change "female” or "male" to "woman” or "man" as appropriate, when used as a noun (see for instance https://apastyle.apa.org/style-grammar-guidelines/bias-free-language/gender).

3. Thank you for stating the following financial disclosure:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

Please state what role the funders took in the study.  If the funders had no role, please state: ""The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.""

If this statement is not correct you must amend it as needed.

Please include this amended Role of Funder statement in your cover letter; we will change the online submission form on your behalf.

4. Thank you for stating the following in the Acknowledgments Section of your manuscript:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

We note that you have provided funding information that is not currently declared in your Funding Statement. However, funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form.

Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows:

“This research was supported by Early-Career Scientists (20K14256) from JSPS to S. N., Early-Career Scientists (19K20387) from JSPS to K.N., Grant-in-Aid for Scientific Research on Innovative Area (17H06344) from JSPS, and by Moonshot R&D (JPMJMS2012) from JST to K.W.”

Please include your amended statements within your cover letter; we will change the online submission form on your behalf.

Additional Editor Comments:

Dear Dr. Namba,

I have now received two reviews of the manuscript entitled The spatio-temporal features of perceived-as-genuine and deliberate expressions (PONE-D-21-33666) that you submitted to PLOS ONE. I was fortunate to secure reviews from two experts in the areas of facial expression analysis and emotional genuineness, both of whom provided strong insight to the strengths and weaknesses of the manuscript.

I read the original manuscript, and then again with comments provided by the reviewers. As you will see, the reviewers found the manuscript to be interesting, and the analytic techniques impressive. The topic is also relevant to the broad readership at PLOS One.

Both reviewers identified the academic merit of the manuscript, however noting that the clarity of the manuscript could be improved. Both reviewers provided guidance on improving the readability, which I encourage you to consider. There was disagreement as to the statistical & methodological rigor of the work, with Reviewer 1 taking issue with some analysis decisions. Both suggested major revisions. As such, I am rejecting the manuscript in its current form, and I invite you to resubmit in the form of major revisions.

You will find the full set of reviewer comments included below. To these I add my own two comments:

1) I would appreciate seeing a clear statement of theoretical contribution of this work. I read your 2021 Scientific Reports article with great interest. However, the 2021 work is only minimally reviewed, and readers may be left unclear as to what this work contributes to our understanding of genuine expressions. I recognize that there are differences with this work – OpenFace vs DeepLabCuts, Happy-Angry vs Surprise, ActionUnits vs landmarks, but these differences are methodological, rather than conceptual. I believe the core contribution relates to the perceptual nature of genuine expressions, but this should be more clearly contrasted from your previous works, along with those in the field.

2) The stimuli presenting Genuine-vs-Deliberate expressions are central to the work, with both perceptual and motion analyses relying on their validity. Yet there is almost no discussion as to how these stimuli were produced, and whether the scenarios would be considered ‘genuine’. This presents a serious methodological oversight which would need to be robustly addressed in your revision.

[Note: HTML markup is below. Please do not edit.]

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: Partly

Reviewer #2: Yes

**********

2. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: No

Reviewer #2: Yes

**********

3. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: Yes

Reviewer #2: Yes

**********

4. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: No

Reviewer #2: Yes

**********

5. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: PRAISE

This manuscript explored the interesting question of whether there are morphological and temporal differences between facial behaviors that are perceived to be genuine vs. deliberate. I liked that it examined perceptions of human facial behaviors rather than computer-generated facial behaviors and agree that this is an important validation step. I also liked the attempt to simultaneously consider morphology and temporal characteristics of the behavioral displays. I also thought the visualizations were nice and appreciated the data and materials being made open.

CONCERNS

1. The manuscript’s readability would be helped if the authors were more consistent in their use of terms. As just one example, multiple different terms are used to refer to the person producing a facial behavior (e.g., sender, encoder) or perceiving it (e.g., observer, decoder, perceiver).

2. The methodological decision to look at onset plus and minus 2 sec creates some standardization but also erodes some of the ecological validity of the stimuli. If temporal characteristics matter for perception, as the authors contend, then why cut them partway through and therefore eliminate the ability of perceivers to see their duration, offset, and other temporal features?

3. Please provide a rationale for making the genuineness vs. deliberate ratings dichotomous. Why not have them rate this quality on a continuum? Forcing a choice between two alternatives throws away valuable information about degree (DeCoster et al., 2009) and introduces (or at least exacerbates) participants’ subjective thresholds as a source of variability.

4. The first paragraph in the Statistical Analysis section (page 6) was very difficult to understand. I think the authors calculated the percentage of participants that rated each stimulus as genuine and then used that quantity to select a subset of the stimuli most consistently rated as genuine vs. deliberate. But the descriptions and some words (“prescribed” and “judgment ratios”) were not adequately clear/explained.

5. Related to concern #4 above, I did not love the methodological decision to analyze 40 stimuli rather than all 180. I think this was done in order to compare behaviors that are perceived-by-most-as-genuine to perceived-by-most-as-deliberate rather than having more ambiguous behaviors. But this selection makes the results much less generalizable. We are not learning about most facial behaviors, we are learning about the 22% least ambiguous ones. If genuineness had been rated dimensionally (as in concern #3), the authors could have regressed it on various temporal features and included all 180 stimuli. If they used a mixed effects model, they also could have explicitly modeled encoder and decoder effects.

6. The authors acknowledge that OpenFace may be incorrect but do not provide any validation evidence such as comparing it to expert FACS coding on a subset of this specific dataset. Because we don’t know how much to trust the AU intensity estimates, we don’t know how much to trust the NMF components and these are really important variables in the paper.

7. Please add some information to the Statistical Analysis section about you will be calculating 99% credible intervals at each value of t and how to interpret the figures (e.g., seeing whether delta=0 falls within that interval). Some of this information is currently in the figure captions only and it is very easy to miss.

8. In addition to plotting the 99% credible intervals at each value of t, I would be interested to see tests done on larger regions of t-space. For example, the authors could calculate the average delta score in the regions from t=0 to t=2000 in intervals of 250 or 500ms. They could also report probability of direction (pd) values for these regions in addition to credible intervals (Makowski et al., 2019).

COMMENTS

1. The claim on page 1 line 50 that perceived-as-genuine expressions have more significant impacts on observer’s behavior than perceived-as-deliberate expressions is a leap that does not follow directly from the evidence presented earlier in the paper.

2. Page 4 would be improved by providing an explicit rationale for the decision to look at anger and happiness rather than other possible emotions.

3. On page 5, please briefly describe what the four emotional scenarios were for happiness and anger.

4. Were there are quality or attention check questions to ensure that the participants were taking the task seriously? I’m not sure if this is a big issue in Japan, but in the US many crowdworkers do a poor job and this can be detected with some basic questions.

REFERENCES

DeCoster, J., Iselin, A.-M. R., & Gallucci, M. (2009). A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349–366. https://doi.org/10/bh86w7

Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019). Indices of effect existence and significance in the Bayesian framework. Frontiers in Psychology, 10. https://doi.org/10/ggfw2j

Reviewer #2: Thank you for inviting me to review this manuscript. The authors investigate spatio-temporal features of facial emotion expressions that help to distinguish perceived-as-genuine and perceived-as-deliberate (i.e., non-authentic) expressions. They do so by first presenting a series of dynamic facial emotion expression videos to 89 Japanese individuals and asking them to determine whether each expression is genuine or deliberate (completed for happy and angry expressions). They then reduced each expression video to its lowest dimensional features (as a function of the emotion and 'perceived as' conditions) using non-negative matrix factorisation. Both perceived-as-genuine and perceived-as-deliberate expressions showed protocol typical spatial action units, whilst deliberate expressions produced multiple facial movements in addition to these. Deliberate expressions also had faster onsets to peak expression and had a greater contribution of mouth opening movements.

The question being investigated here is novel, interesting and important. The techniques utilised to answer this question are in my opinion strong and in line with a number of recent studies applying dimensionality reduction techniques to facial emotion expressions. I commend the authors for this. I find the results very interesting and I also think they did a good job to put their findings in context as well as discuss the limitations (e.g., cultural differences that may exist and require further attention.

I have some comments on the write-up (mostly focused on the introduction) that I believe would strengthen the manuscript if rectified:

Introduction:

1) I feel the introduction could be tightened up a little bit so it really 'tells the story' for your reader to follow as to the rational for the current study. Some parts I struggled to follow the relevance of. For example, I wasn't sure I followed how the study is specifically related to the affective pragmatics theory and why this needs to be linked to an 'evolutionary account' that really had very little explanation. Perhaps you can build a story for the studies importance without these theories, especially as you don't refer back to this in the discussion, so I'm not sure exactly how relevant it is. Or alternatively, tighten up the explanation and refer back to it in the discussion.

2) I think your design is great, whereby you are able to determine what spatio-temporal features are important in the human perception or labelling of genuine vs deliberate (as opposed to determining yourselves what a genuine or deliberate expression looks like) and I agree this is in line with the work by Rachel Jack and colleagues. However, I think you could make this point and emphasise the benefit of this kind of approach clearer for the reader. You could tighten up the explanation of Jack and co.'s method.

3) My colleagues and I have investigated temporal features (mean speed) of happy, angry and sad expressions in posed and spontaneous contexts (Sowden et al., 2021; https://psycnet.apa.org/fulltext/2021-22419-001.html) which might also fit well into your introduction and/or discussion to support differences here. We found (in Experiment 1C) that speed of mouth opening and widening actions helped to differentiate between happy, angry and sad expressions only for posed and not spontaneous expressions, whilst speed of eyebrow widening movements was capable of differentiating these emotions for both posed and spontaneous expressions. This to me seems a nice finding alongside your findings, whilst your design has the nice element of looking at timecourses as opposed to our mean speed over entire expression period.

Methods

1) I think the methods section would benefit from a little more explanation of what the NMF dimensionality reduction technique is and what it is essentially doing, for readers who do not know about this technique. On this topic, the authors might be interested in (and may wish to cite) an interesting paper by Chiovetti et al. (2018; https://jov.arvojournals.org/article.aspx?articleid=2678770) who seek to reduce various facial expressions down to their lowest dimensional organisation (regarding temporal properties of expressions) whereby emotion can be determined. They also compare NMF and a Fourier-based Anechoic Demixing Algorithm in doing so.

2) Can you be slightly clearer in your methods by telling the reader that the way you arrived at 4840 data points is by taking each frame (121) in each of the 40 videos as a data point?

3) Could you clarify how exactly you made all the videos to be of the same length, this wasn't totally clear to me?

Small changes:

Page 3, line 85: I think you should refer to 'emotion categories' as opposed to mental representations to be clearer here for the reader.

Page 6, line 159: Was the ISI 'about' 300 ms or 'exactly' 300 ms? I.e., did you have a variable ISI or was this always 300ms?

Page 9, line 240: Remove the word 'mean' or 'refer to'.

**********

6. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jeffrey Girard

Reviewer #2: No

[NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.]

While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/. PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org. Please note that Supporting Information files do not need this step.

17 May 2022

Additional Editor Comments:

Dear Dr. Namba,

I have now received two reviews of the manuscript entitled The spatio-temporal features of perceived-as-genuine and deliberate expressions (PONE-D-21-33666) that you submitted to PLOS ONE. I was fortunate to secure reviews from two experts in the areas of facial expression analysis and emotional genuineness, both of whom provided strong insight to the strengths and weaknesses of the manuscript.

I read the original manuscript, and then again with comments provided by the reviewers. As you will see, the reviewers found the manuscript to be interesting, and the analytic techniques impressive. The topic is also relevant to the broad readership at PLOS One.

Both reviewers identified the academic merit of the manuscript, however noting that the clarity of the manuscript could be improved. Both reviewers provided guidance on improving the readability, which I encourage you to consider. There was disagreement as to the statistical & methodological rigor of the work, with Reviewer 1 taking issue with some analysis decisions. Both suggested major revisions. As such, I am rejecting the manuscript in its current form, and I invite you to resubmit in the form of major revisions.

You will find the full set of reviewer comments included below. To these I add my own two comments:

1) I would appreciate seeing a clear statement of theoretical contribution of this work. I read your 2021 Scientific Reports article with great interest. However, the 2021 work is only minimally reviewed, and readers may be left unclear as to what this work contributes to our understanding of genuine expressions. I recognize that there are differences with this work – OpenFace vs DeepLabCuts, Happy-Angry vs Surprise, ActionUnits vs landmarks, but these differences are methodological, rather than conceptual. I believe the core contribution relates to the perceptual nature of genuine expressions, but this should be more clearly contrasted from your previous works, along with those in the field.

Thank you for your comments. We have added a new paragraph with a more detailed description of the contribution of this work (Page 17, Lines 412–421).

2) The stimuli presenting Genuine-vs-Deliberate expressions are central to the work, with both perceptual and motion analyses relying on their validity. Yet there is almost no discussion as to how these stimuli were produced, and whether the scenarios would be considered ‘genuine’. This presents a serious methodological oversight which would need to be robustly addressed in your revision.

Following to the editor’s recommendation, we have added story examples (Pages 5-6, Lines 150–153) and a more detailed explanation of and a description of the limitations to our selection of facial database (Page 5, Lines 138–144; Page 18, Lines 441–444).

We sincerely appreciate your reading and consideration of our paper.

Comments to the Author

We are grateful for the reviewers’ excellent and extremely helpful comments. We have addressed all of the issues that the reviewers have highlighted, and we believe that the manuscript has been considerably improved as a result of these changes. Please let us know if further changes are required. We will be more than happy to alter the manuscript further.

Reviewer #1:

PRAISE

This manuscript explored the interesting question of whether there are morphological and temporal differences between facial behaviors that are perceived to be genuine vs. deliberate. I liked that it examined perceptions of human facial behaviors rather than computer-generated facial behaviors and agree that this is an important validation step. I also liked the attempt to simultaneously consider morphology and temporal characteristics of the behavioral displays. I also thought the visualizations were nice and appreciated the data and materials being made open.

CONCERNS

1. The manuscript’s readability would be helped if the authors were more consistent in their use of terms. As just one example, multiple different terms are used to refer to the person producing a facial behavior (e.g., sender, encoder) or perceiving it (e.g., observer, decoder, perceiver).

Thank you for this comment. We have changed the terms to make them more consistent way (e.g., Page 1, Lines 48–49; Page 2, Lines 53-54…). We hope that these changes will help increase readability.

2. The methodological decision to look at onset plus and minus 2 sec creates some standardization but also erodes some of the ecological validity of the stimuli. If temporal characteristics matter for perception, as the authors contend, then why cut them partway through and therefore eliminate the ability of perceivers to see their duration, offset, and other temporal features?

We agree with Reviewer #1 in that the standardized onset and offset of facial expressions could potentially undermine ecological validity. Therefore, we must keep in mind that the conclusions drawn from our observations could be limited. Nevertheless, we needed to instruct the models to intentionally produce facial expressions within a certain time range to make it easier to compare perceived genuine and deliberate expressions at the expense of the natural and spontaneous time course of facial expressions. We believed that a certain degree of standardization (i.e., looking at onset plus and minus 2 sec) is desirable when presenting these facial datasets. However, as the reviewer mentioned, traditionally it is more desirable to consider offset as well (e.g., Schmidt et al., 2006: Movement Differences between Deliberate and Spontaneous Facial Expressions: Zygomaticus Major Action in Smiling), and recent studies have suggested that offset is important in determining authenticity (Horic-Aselin et al., 2020).

To clarify these points, we added the description of the potential limitations of our method in the Discussion section (Page 21, Lines 441–444) and provided more detailed explanation of the method of video recording in the Method section (Page 5, Lines 138–144).

3. Please provide a rationale for making the genuineness vs. deliberate ratings dichotomous. Why not have them rate this quality on a continuum? Forcing a choice between two alternatives throws away valuable information about degree (DeCoster et al., 2009) and introduces (or at least exacerbates) participants’ subjective thresholds as a source of variability.

Thank you for your comments. As you suggest, it would be more conservative to measure this authenticity as a continuous rating on two axes, genuine and deliberate (Dawel et al., 2017). Consequently, a better approach would be to provide a rating scale instead of a binary judgement. We have added this limitation to the Discussion section and cited your recommended work (Page 19, Lines 453-458).

4. The first paragraph in the Statistical Analysis section (page 6) was very difficult to understand. I think the authors calculated the percentage of participants that rated each stimulus as genuine and then used that quantity to select a subset of the stimuli most consistently rated as genuine vs. deliberate. But the descriptions and some words (“prescribed” and “judgment ratios”) were not adequately clear/explained.

Following the reviewer’s suggestion, we have modified our description for reader clarity (Pages 8–9, Lines 208–223). In addition to modifying this description, we have made major changes to the study actions; please refer to the response to the following comment (Comment #5).

5. Related to concern #4 above, I did not love the methodological decision to analyze 40 stimuli rather than all 180. I think this was done in order to compare behaviors that are perceived-by-most-as-genuine to perceived-by-most-as-deliberate rather than having more ambiguous behaviors. But this selection makes the results much less generalizable. We are not learning about most facial behaviors, we are learning about the 22% least ambiguous ones. If genuineness had been rated dimensionally (as in concern #3), the authors could have regressed it on various temporal features and included all 180 stimuli. If they used a mixed effects model, they also could have explicitly modeled encoder and decoder effects.

Thank you for this insightful comment. Following the reviewer’s suggestion, we used as many stimuli as possible for the analysis. We sincerely appreciate your comment and apologize for the discrepancy in the revised paper caused by conducting a detailed analysis from scratch (N = 180 → 148). We have drastically altered our statistical approach using the complete dataset.

First, we used all of the data for anger and happiness that the current study used (N = 128) and confirmed that the same NMF spatial patterns were extracted as reported in the first draft. This can be interpreted as indicating the robustness of the NMF data obtained in this study.

Regarding the relationship between the decoders’ judgments of genuineness and identified NMF spatial patterns, we added a new mixed model which had explicitly modeled encoder and decoder effects. This logistic regression analysis provided us with results that are more easily interpretable for the spatio-temporal features of perceived-as-deliberate expressions (Page 8, Lines 198–207; Page 11, Lines 267–273; Page 13, Lines 309–313).

For the temporal features, we added “ambiguous” as a new condition for facial expressions that fell into the rating between as the intermediate between perceived-as-genuine and perceived-as-deliberate. By splitting δ into two, we succeeded in finding a generalization effect of increasing data and in separating the perceived-as-genuine effect from the perceived-as-deliberate effect (Pages 9-10, Lines 228–245).

We hope that our statistical approach has been improved drastically thanks to the reviewers’ detailed comments.

6. The authors acknowledge that OpenFace may be incorrect but do not provide any validation evidence such as comparing it to expert FACS coding on a subset of this specific dataset. Because we don’t know how much to trust the AU intensity estimates, we don’t know how much to trust the NMF components and these are really important variables in the paper.

Reviewer #1 is right. Although our previous study showed that OpenFace AU detection for Japanese also performs similarly to manual coding by a certificated FACS coder (Namba et al., 2021; r = 0.8), we have added further details to the method and limitation sections with the addition of this information (Page 7, Lines 184-186; Page 19, Lines 459-469). Furthermore, by adding new analyses (thanks to Comment #5), it is clear that the results of NMF are reliable, at least in the sense that they contribute to the decoder's judgment (Figs 2, 4). Thank you.

7. Please add some information to the Statistical Analysis section about you will be calculating 99% credible intervals at each value of t and how to interpret the figures (e.g., seeing whether delta=0 falls within that interval). Some of this information is currently in the figure captions only and it is very easy to miss.

Following the reviewer’s suggestion, we have added further explanations of credible intervals (Page 8, Lines 206–207; Page 10, Lines 244–245).

8. In addition to plotting the 99% credible intervals at each value of t, I would be interested to see tests done on larger regions of t-space. For example, the authors could calculate the average delta score in the regions from t=0 to t=2000 in intervals of 250 or 500ms. They could also report probability of direction (pd) values for these regions in addition to credible intervals (Makowski et al., 2019).

Thank you very much for your thoughtful comment. We have calculated the average delta score at intervals of 500 ms. Following this, we extracted the 99% credible intervals and probability of direction (pd) values for these regions. For readability, we have provided this information in the Supplemental Tables.

COMMENTS

1. The claim on page 1 line 50 that perceived-as-genuine expressions have more significant impacts on observer’s behavior than perceived-as-deliberate expressions is a leap that does not follow directly from the evidence presented earlier in the paper.

We have modified this paragraph to clarify the intended meaning (Pages 1–2, Lines 50–55).

2. Page 4 would be improved by providing an explicit rationale for the decision to look at anger and happiness rather than other possible emotions.

We have added a description identifying the rationale for looking at anger and happiness (Pages 3-4, Lines 102–104).

3. On page 5, please briefly describe what the four emotional scenarios were for happiness and anger.

As the reviewer noted, we have added the example of stories (Pages 5-6, Lines 150–153).

4. Were there are quality or attention check questions to ensure that the participants were taking the task seriously? I’m not sure if this is a big issue in Japan, but in the US many crowdworkers do a poor job and this can be detected with some basic questions.

The experiment we employed this time took the form of contracting and hiring each person over time, so we expect the results to be similar in nature to those of a face-to-face experiments done online.

(There will be some degree of responsibility in that each participant will be linked to an answer and an ID, rather than just distributing links, as tends to happen in general online experiments.)

However, we acknowledged that response bias may be inevitable for online participants, and it is important to include such item checks. For this reason, we have provided a justification for Crowdworks Participants (Page 5, Lines 126–127) and added this information the discussion (Pages 18-19, Lines 449–453).

REFERENCES

DeCoster, J., Iselin, A.-M. R., & Gallucci, M. (2009). A conceptual and empirical examination of justifications for dichotomization. Psychological Methods, 14(4), 349–366. https://doi.org/10/bh86w7

Makowski, D., Ben-Shachar, M. S., Chen, S. H. A., & Lüdecke, D. (2019). Indices of effect existence and significance in the Bayesian framework. Frontiers in Psychology, 10. https://doi.org/10/ggfw2j

Reviewer #2: Thank you for inviting me to review this manuscript. The authors investigate spatio-temporal features of facial emotion expressions that help to distinguish perceived-as-genuine and perceived-as-deliberate (i.e., non-authentic) expressions. They do so by first presenting a series of dynamic facial emotion expression videos to 89 Japanese individuals and asking them to determine whether each expression is genuine or deliberate (completed for happy and angry expressions). They then reduced each expression video to its lowest dimensional features (as a function of the emotion and 'perceived as' conditions) using non-negative matrix factorisation. Both perceived-as-genuine and perceived-as-deliberate expressions showed protocol typical spatial action units, whilst deliberate expressions produced multiple facial movements in addition to these. Deliberate expressions also had faster onsets to peak expression and had a greater contribution of mouth opening movements.

The question being investigated here is novel, interesting and important. The techniques utilised to answer this question are in my opinion strong and in line with a number of recent studies applying dimensionality reduction techniques to facial emotion expressions. I commend the authors for this. I find the results very interesting and I also think they did a good job to put their findings in context as well as discuss the limitations (e.g., cultural differences that may exist and require further attention.

I have some comments on the write-up (mostly focused on the introduction) that I believe would strengthen the manuscript if rectified:

Introduction:

1) I feel the introduction could be tightened up a little bit so it really 'tells the story' for your reader to follow as to the rational for the current study. Some parts I struggled to follow the relevance of. For example, I wasn't sure I followed how the study is specifically related to the affective pragmatics theory and why this needs to be linked to an 'evolutionary account' that really had very little explanation. Perhaps you can build a story for the studies importance without these theories, especially as you don't refer back to this in the discussion, so I'm not sure exactly how relevant it is. Or alternatively, tighten up the explanation and refer back to it in the discussion.

Thank you very much for your thoughtful comments. We agree with this suggestion, and we have attempted to tighten up our introduction and omit these theories (Pages 1–2, Lines 47–57), referring back to this in the discussion section (Page 17, Lines 412–421). We hope that this change has improved the readability of the manuscript. Thanks again.

2) I think your design is great, whereby you are able to determine what spatio-temporal features are important in the human perception or labelling of genuine vs deliberate (as opposed to determining yourselves what a genuine or deliberate expression looks like) and I agree this is in line with the work by Rachel Jack and colleagues. However, I think you could make this point and emphasise the benefit of this kind of approach clearer for the reader. You could tighten up the explanation of Jack and co.'s method.

Thank you for your suggestion. In response to your comment, we have tightened up our presentation of Jack and her colleague’s method, and we have emphasized the specific benefit our approach brings relative to theirs in this section (Page 3, Lines 78–88)

3) My colleagues and I have investigated temporal features (mean speed) of happy, angry and sad expressions in posed and spontaneous contexts (Sowden et al., 2021; https://psycnet.apa.org/fulltext/2021-22419-001.html) which might also fit well into your introduction and/or discussion to support differences here. We found (in Experiment 1C) that speed of mouth opening and widening actions helped to differentiate between happy, angry and sad expressions only for posed and not spontaneous expressions, whilst speed of eyebrow widening movements was capable of differentiating these emotions for both posed and spontaneous expressions. This to me seems a nice finding alongside your findings, whilst your design has the nice element of looking at timecourses as opposed to our mean speed over entire expression period.

Thank you for mentioning these important reports in the literature. We have cited these works in the revised manuscript (Page 2, Lines 69–71; Page 16, Lines 382–388; Page 16, Lines 393–395). We are honored that the relationship with the proposed study has enriched our contribution.

Methods

1) I think the methods section would benefit from a little more explanation of what the NMF dimensionality reduction technique is and what it is essentially doing, for readers who do not know about this technique. On this topic, the authors might be interested in (and may wish to cite) an interesting paper by Chiovetti et al. (2018; https://jov.arvojournals.org/article.aspx?articleid=2678770) who seek to reduce various facial expressions down to their lowest dimensional organisation (regarding temporal properties of expressions) whereby emotion can be determined. They also compare NMF and a Fourier-based Anechoic Demixing Algorithm in doing so.

Thank you for noticing these important reports from the literature. We have added a simpler explanation for the NMF approach and added information on the Chiovetto’s work in the revised manuscript (Page 7, Lines 190–197).

2) Can you be slightly clearer in your methods by telling the reader that the way you arrived at 4840 data points is by taking each frame (121) in each of the 40 videos as a data point?

We have modified this paragraph to clarify our intention (Page 8, Lines 216–217).

3) Could you clarify how exactly you made all the videos to be of the same length, this wasn't totally clear to me?

In response to this comment, we added information about how we made all the videos the same length (Page 5, Lines 138–146).

Small changes:

Page 3, line 85: I think you should refer to 'emotion categories' as opposed to mental representations to be clearer here for the reader.

Accordingly, we have added a description in this paragraph (Page 3, Lines 79–81).

Page 6, line 159: Was the ISI 'about' 300 ms or 'exactly' 300 ms? I.e., did you have a variable ISI or was this always 300ms?

Page 9, line 240: Remove the word 'mean' or 'refer to'.

We have added the term exactly and removed the word mean. Thank you for noticing these points (Page 6, Line 174, Page 12, Lines 294–297).

Submitted filename: Response_to_Reviewers.docx

Click here for additional data file.

23 Jun 2022

The spatio-temporal features of perceived-as-genuine and deliberate expressions

PONE-D-21-33666R1

Dear Dr. Namba,

We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Please see the minor suggested comments of R2, which you may wish to incorporate prior to proofing.

Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication.

An invoice for payment will follow shortly after the formal acceptance. To ensure an efficient process, please log into Editorial Manager at http://www.editorialmanager.com/pone/, click the 'Update My Information' link at the top of the page, and double check that your user information is up-to-date. If you have any billing related questions, please contact our Author Billing department directly at authorbilling@plos.org.

If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org.

Kind regards,

Steven R. Livingstone

Academic Editor

PLOS ONE

Additional Editor Comments (optional):

Please see the minor suggested comments from R2 prior to your proofing.

Reviewers' comments:

Reviewer's Responses to Questions

Comments to the Author

1. If the authors have adequately addressed your comments raised in a previous round of review and you feel that this manuscript is now acceptable for publication, you may indicate that here to bypass the “Comments to the Author” section, enter your conflict of interest statement in the “Confidential to Editor” section, and submit your "Accept" recommendation.

Reviewer #1: All comments have been addressed

Reviewer #2: All comments have been addressed

**********

2. Is the manuscript technically sound, and do the data support the conclusions?

The manuscript must describe a technically sound piece of scientific research with data that supports the conclusions. Experiments must have been conducted rigorously, with appropriate controls, replication, and sample sizes. The conclusions must be drawn appropriately based on the data presented.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

3. Has the statistical analysis been performed appropriately and rigorously?

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

4. Have the authors made all data underlying the findings in their manuscript fully available?

The PLOS Data policy requires authors to make all data underlying the findings described in their manuscript fully available without restriction, with rare exception (please refer to the Data Availability Statement in the manuscript PDF file). The data should be provided as part of the manuscript or its supporting information, or deposited to a public repository. For example, in addition to summary statistics, the data points behind means, medians and variance measures should be available. If there are restrictions on publicly sharing data—e.g. participant privacy or use of data from a third party—those must be specified.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

5. Is the manuscript presented in an intelligible fashion and written in standard English?

PLOS ONE does not copyedit accepted manuscripts, so the language in submitted articles must be clear, correct, and unambiguous. Any typographical or grammatical errors should be corrected at revision, so please note any specific errors here.

Reviewer #1: (No Response)

Reviewer #2: Yes

**********

6. Review Comments to the Author

Please use the space provided to explain your answers to the questions above. You may also include additional comments for the author, including concerns about dual publication, research ethics, or publication ethics. (Please upload your review as an attachment if it exceeds 20,000 characters)

Reviewer #1: (No Response)

Reviewer #2: I am happy with responses to my comments and associated changes. Just three very minor comments remain:

1) When discussing the Sowden et al paper in the introduction, for clarity I would change the sentence to read: "Sowden et al demonstrated, using facial landmarks, that speed of facial movements differentiates deliberate expressions of anger, happiness and sadness."

2) Line 106: delete the words 'as to'.

3) Line 169-170: change sentence to read "If the participants responded with no questions, the main trials began."

**********

7. PLOS authors have the option to publish the peer review history of their article (what does this mean?). If published, this will include your full peer review and any attached files.

If you choose “no”, your identity will remain anonymous but your review may still be made public.

Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy.

Reviewer #1: Yes: Jeffrey Girard

Reviewer #2: No

**********

7 Jul 2022

PONE-D-21-33666R1

The spatio-temporal features of perceived-as-genuine and deliberate expressions

Dear Dr. Namba:

I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now with our production department.

If your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information please contact onepress@plos.org.

If we can help with anything else, please email us at plosone@plos.org.

Thank you for submitting your work to PLOS ONE and supporting open access.

Kind regards,

PLOS ONE Editorial Office Staff

on behalf of

Dr. Steven R. Livingstone

Academic Editor

PLOS ONE