Virtual eye region: development of a realistic model to convey emotion.

The human eyes and their surrounding features are capable of conveying an array of emotional and social information through expressions. Producing virtual human eyes which are able to communicate these complex mental states continues to be a challenging research topic in computer graphics (CG) as subtle inaccuracies can be the difference between realistic and uncanny. With the recent emergence of virtual customer service agents, the demand for expressive virtual eyes is increasing. One essential question that remains to be answered is: Can virtual human eyes effectively transmit emotion? Through a combination of 3D scanning and manual hand modelling techniques, we developed an efficient pipeline to realise a virtual model of the human eye area that displays real-world characteristics. From this model eye expression renders of the six basic emotions, anger, disgust, fear, happiness, sadness and surprise were generated (Ekman et al., 1969). The perceptual quality of the model was evaluated by showing respondents from two age groups the six eye expressions renders and corresponding real-world photos. Respondents then judged which of the six emotions best described each image. Our findings indicate a clear relationship between the recognition levels for both photographic and virtual stimuli plus a significant level of emotional perception was found for the virtual eye expressions of sadness and anger. This research of human cognition and CG is a starting point for investigating the use of artificial human eye expressions as an effective research tool in the perceptual community.

Introduction

Psychological science has long speculated about how human eyes display important emotional cues to transmit our state of mind. This subject has fascinated researchers for centuries, at least since the publication of The Expression of Emotions in Man and Animals [1]. The human eye is made unique amongst primates by the amount of exposed eye white (sclera) in contrast to the pigmentation of the iris [2]. In fact, further studies found that human eyes had three times more visible sclera than in orangutans when looking straight ahead [3]. Researchers hypothesise that this unique colouration of the human eye is an evolving adaptation to enhance the gaze signal, making the human eye a tool for communication [4]. From an early age, the eye area is an essential means for human interaction [5]. We naturally expect a living human to display a variety of social and emotional information [6]. These emotions are grounded in the expressive visual cues of the eyes and the eye region [7], in fact, the eyes are an exceptionally accurate source of emotional information even without the context of other facial features [8].

With the technological progress that computer graphics (CG) and artificial intelligence (AI) has made over the last decade, it is predictable that virtual customer service agents with the ability to detect and react to human emotion will soon be commonplace. This will create a strong demand for expressive virtual eyes, an area which has received little attention from the CG community - research projects have tended to be directed towards other features such as hair, teeth, and skin [9, 10, 11]. If these digital avatars are to be accepted into our digital culture and enhance communication it is important to understand if humans could perceive emotion from within virtual eyes. Understanding gained from the experiment would have implications for the fields of CG and psychology. For example, if a notable level of emotional recognition can be shown within the virtual model, then this would indicate that CG could be a valid means of research for psychologists to study human eye expressions. Furthermore, knowledge gained from the development of the virtual model would help CG artists to create more technically realistic eyes and factors relating to the subtle visual cues which are present in each emotion may also help CG artists to generate more expressive CG eyes which in turn are more believable. The goal of this paper is to take a few small steps in demonstrating that virtual eyes outside of the creative industries can be used in psychological research to study the human perception of emotion through eye expressions.

Related work

A broad amount of research has been done on various aspects of the eye in both CG and psychological science. Early experiments in CG used a simple sphere to depict the human eye [12]. Later studies use image-based approaches to reconstruct the human iris digitally, including its light scattering features from a single eye photograph. The result was then integrated within a 3D eye model [13]. The rise of specialised graphics processing units (GPUs) and real-time rendering then gave designers the ability to add interactive features to their 3D eye models such as automatic pupil adaption to the illumination level within a virtual environment [14]. More recently an innovative approach was developed to capture the visible part of the human eye as a 3d model in high detail - this investigation revealed that each eye is unique in both form and texture [15]. Unfortunately, the acquisition process was long and awkward therefore a revised light-weight method was developed two years later which produces a realistic 3D eye model from only a single high-resolution photo [16].

Research in psychological science supports the notion that the eye area is essential when recognising complex mental states which go beyond the basic emotions [8]. More current research demonstrates that human eyes convey a variety of complex mental states through expressive eye features, for example, brow distance and eye aperture for fear and brow slope and nasal wrinkles for disgust [7]. Comparison of emotions between real and digital faces have been carried out before with varying results [17, 18]. Both studies found that recognition of emotional facial expressions in avatars is comparably less when compared to humans. However, other research has reported similar recognition levels [19, 20]. The difference between these experiments could be attributed to the varying visual quality of the virtual stimuli used. Furthermore, these experiments focused on all facial features being processed as one entity (referred to as holistic face processing) rather than piecemeal processing on separate face parts.

Our work focuses solely on developing a realistic (pre-rendered) virtual model of the eye region which can display the subtle cues necessary to transmit an emotional state to a viewer. An area of research in human visual perception which is overlooked.

Design

Subject

For the creation of the virtual stimuli to be used in the experiment, a healthy female subject of 22 years old was selected as previous studies have shown that overall women display greater emotional expression than men, especially for positive emotions [21].

Eyeball model

Our anatomically inspired eyeball model was built as close as possible to real-world scale using dimensions of the average adult human eye: 24.2mm (transverse) x 23.7mm (sagittal) x 22.0–24.8mm (axial) [22]. Scale plays a crucial role in CG as many features in modern applications rely on the real-world scale to render results correctly. The model consists of three meshes (see Fig. 1) which include the cornea, sclera/iris and pupil/lens.

Fig. 1

(a) Our eye model includes the cornea, sclera, iris and pupil/lens (b) A sample rendering is shown.

The outer mesh represents the cornea which is transparent, reflective and refractive. To imitate real-world values the correct index of refraction (IOR) is used (n = 1.376). A fundamental property produced by the cornea is light scattering which can affect many parts of the ocular tissue underneath [23]. To recreate this effect within the eyeball model Sub-Surface Scattering (SSS) is used to simulate the light entering the eye and scattering beneath the surface. This effect was calculated using the 3D rendering application Arnold which uses a brute-force raytracing method to produce more natural-looking results than other diffusion-based approximations [24].

The inner mesh combines the sclera (the white of the eye) and iris. The texture map for the sclera and iris was based on a number of high-resolution photographs of the eye with the subject looking in different directions, without polarization. These photographs were composited together with image manipulation software to generate the final 4K colour texture. The sclera's surface imperfections were created by applying a normal map of the conjunctiva (extracted from the final colour texture) to the Normal (Bump) attribute within the sclera/iris shader. To simulate the aqueous humor (n = 1.336) between the pupil and cornea a Coat attribute (which replicates a dielectric) was applied on top of the sclera/iris shader. The pupil/lens mesh is created from a flattened sphere and distorted to mirror the form shown in eye structure diagrams.

Eye region geometry

For the eyeball model rendering to be considered realistic, it must also feature realistic neighbouring facial detail. To capture these features accurately we used an advanced laser scanner (HP-L-20.8) developed by Hexagon Metrology which allows rapid capture of free form surfaces. High-resolution photography of the subject was also utilised to create 4K photo-realistic skin textures. The subject posed with a neutral facial expression as this would give reference scan geometry from which a retopologised base mesh could be later generated. Even though a Class 2 laser can be considered safe due to a human's blink reflex optic safety was ensured by the subject closing their eyes for the duration of the scan. The closed eyelids could be simply removed and replaced with our own eyeball model. The scanner captures the subjects face by digitising multiple points using laser triangulation resulting in a dense point cloud which is used to construct a triangulated mesh. The process took about a minute to capture the facial features in the required detail and produced a triangulated mesh of about 1 million vertices.

Although the triangulated mesh produced by the scanning process has high-resolution topology it also has inefficient edge flow (see Fig. 2) which cannot be easily controlled when changing the shape of a mesh, for example creating facial expressions. We, therefore, retopologised the scan geometry using manual hand modelling techniques to create a more efficient (lower resolution) mesh where the edges flow around the natural contours of facial features. This allows for a more realistic mesh deformation that resembles actual skin tissue and muscles [25]. Through the retoplogisation process the new efficient mesh losses much of the resolution and subtle skin details of the original scan geometry. These were replaced by subdividing the new mesh and applying a normal map (generated from the 4K diffuse colour textures) to the skin shader. The final retopologised and subdivided neutral face mesh consists of 39,630 vertices. The additional main features within the eye area such as the eyelashes, eyebrows, canthus, and tear line was realised as 3D geometry using standard hand modelling methods (see Fig. 3).

Fig. 2

(a) Final scan mesh (b) Close-up of scanned mesh with inefficient edge flow (c) Retopologised mesh.

Fig. 3

(a) Progression from low resolution retopologised mesh to final subdivided mesh with added features (b) Canthus and tear line geometry in blue (c) Wireframe of eyelashes and eyebrow details.

Posing the virtual model

The six basic emotions (anger, disgust, fear, happiness, sadness, and surprise) [26] were realised by visually referencing and transferring the micro facial cues indicated by the Facial Action Coding System (FACS) onto the virtual base mesh [27]. The FACS is a research tool for measuring facial muscle movements zin correspondence to a displayed emotion. It has become a commonly used tool by animators to achieve accurate facial expressions of emotion and help with facial animation posing and rigging [28]. For further visual aid, the subject was also photographed displaying the six emotions. To ensure the results were as natural as possible the subject received visual and written instructions from the FACS of the six different expressions in advance and was asked to rehearse prior to the photoshoot. It was emphasized that the subject should express the emotion in a way which felt natural and to make the expression distinct and clear. Two-Point diffuse lighting setup was used throughout to guarantee that the subject was evenly lit with minimal strong highlights and shadows.

Technically the virtual eye expressions were created using morph targets or blend shapes as previous studies into 3D facial expressions have shown that this approach gives the artist more manual control than alternative facial expression methods [29]. Furthermore, this method also stimulates more creative input into the creation of the visual cues [30]. Fig. 4 provides an example of the micro-expressions identified by the FACS for the emotion of sadness and the corresponding render of the posed eye region morph.

Fig. 4

(a) Micro expressions of sadness identified by the FACS and illustrated by the subject (b) Eye region render with mirrored micro expressions.

For this study, a set of six eye expression morphs have been created using this pipeline but it could be a feasible process to produce a larger range of eye region morphs if the study was to be extended. As can be seen in Fig. 5, once the eye model and eye region base mesh has been generated each required eye region morph can be posed from this foundation. The final virtual eye region renders are shown in Fig. 6.

Fig. 5

Our eye region model pipeline.

Fig. 6

Samples of the six rendered eye expressions created using the model pipeline (a). Anger (b). Disgust (c). Fear (d). Happiness (e). Sadness (f). Surprise.

Photo stimuli

The photos of the eye expressions used in the online survey were obtained from The Karolinska Directed Emotional Faces (KDEF) [31]. For this study, we selected six photos from the database (BF29ANS/AF27DIS/AF02AFS/BF06HAS/AF07SAS/BF08SUS), one for each emotion and cropped them to the eye region. To ensure a degree of continuity between the photos and renders, the photos chosen were matched for gender, age, and race to the subject used in the creation of the virtual model. Technically the renders are generated at the same resolution as the photos, about 850 × 363 pixels and also colour balanced to give a similar visual fidelity (see Fig. 7).

Fig. 7

(a) Original photo from KDEF: AF07SAS (b) Photo cropped to eye region (c) Colour balanced virtual eye region morph.

Experimental

Apparatus

Respondents completed the investigation on the internet via an online survey. The length of the survey's duration, viewing distance and hardware were uncontrolled. All of the completed surveys lasted under 3.5 min.

Method

Respondents totalled 106 consisting of 31 experienced and 75 laypersons. Respondents were coded as ‘experienced’ if they indicated some competence in visual effects, 3d animation or CG and ‘layperson’ if no previous knowledge was held. The final results exclude the data of the 31 experienced respondents to eliminate any bias which could result from their exposure to realistic, digital humans. The age of the layperson respondents covered a wide range and was divided into two different age groups: under-55 vs. over-55. The under-55 group amounted to 60 respondents (80.00% of the data set) and the over-55 group to 15 respondents (20.00% of the data set). All respondents consented to the use of their information in this study by viewing a statement and completing the online survey. After reading instructions and confirming age group and experience level respondents were shown the 12 eye expressions images (6 renders and 6 photos) separately and in a random order which alternated between render/photo or photo/render to retain even distribution of media type. No time limit was imposed to reduce carryover and interference plus respondents could continue to the next image whenever they felt ready by clicking a ‘next’ button. For every image each respondent was asked to assign an emotion which best describes that image from the following list: (a) Anger, (b) Disgust, (c) Fear, (d) Happiness, (e) Sadness, and (f) Surprise.

Results

Accuracy comparison: Comparable accuracy rates between the photographic and virtual eye regions for each emotion (anger, disgust, fear, happiness, sadness, and surprise) are defined in Table 1. In all instances, the photographic stimuli achieved better recognition rates than their virtual counterpart. One sample t-tests between percentages revealed significant differences in accuracy (p < .05) for the emotions of anger, disgust, happiness and surprise. Sadness attained the closest recognition rate between both media types.

Table 1

Comparable accuracy rates in % for photographic and virtual stimuli.

Emotion	Photo	Virtual	p-value
Anger	89.33	57.33	.020*
Disgust	53.33	30.66	.030*
Fear	57.33	37.33	>.5
Happiness	74.66	46.66	.026*
Sadness	77.33	73.33	>.5
Surprise	85.33	49.33	.006**

A lower p-value indicates a higher significance of recognition disparity between both media types. Bold is used to show that the p-value calculated for that emotion is under .05 which indicates a statistical significance.

Accuracy rates summarised in Table 1 are further illustrated in Fig. 8. Presenting the data in a line graph format shows a clear correlation between the recognition levels of both stimuli, for example, both trend lines illustrate a notable ‘dip’ in recognition for the emotions of disgust and fear and higher levels of recognition for anger, happiness, and sadness. The almost constant difference in outcome for anger, disgust, fear, and happiness possibly suggest that the micro-expressions for most of the virtual eye region renders were not quite visually strong enough.

Fig. 8

Comparable accuracy rates with polynomial (R2 = 0.2) trend lines.

Age range accuracy: The under-55 group achieved a mean accuracy rate of 72.50% (SD = 30.47%) for the photo stimuli and 46.11% (SD = 68.55%) for the virtual stimuli. In contrast the over-55 group attained a mean accuracy rate of 75.55% (SD = 27.50%) for the photo stimuli and 52.22% (SD = 72.31%) for the virtual stimuli (see Table 2/Fig. 9). These figures suggest that from the two age categories respondents above the age of 55 tended to recognise emotion marginally better than younger respondents in both photographic and virtual categories. However, a comparison of means between age groups - photo stimuli (p=0.63) and virtual stimuli (p=0.51) reveals that these differences are not of a significant degree. Another notable comparison is the standard deviation similarity of the mean accuracy across both age groups. The photographic stimuli datasets show a standard deviation of under half that of its virtual counterparts. This indicates that respondents of all ages tended to achieve a more consistent perception score with the photographic emotions whereas the recognition level of the virtual emotions was more likely to be dispersed pointing towards a more distinct lower or higher result and perception level.

Table 2

Percentage accuracy ratings for each group viewing both photographic and virtual stimuli.

Ratings (%)
Emotion	Anger	Disgust	Fear	Happiness	Sadness	Surprise
Under-55 Photo
Anger	90.00	46.66	13.33	3.33	5.00	0
Disgust	10.00	53.33	20.00	13.33	5.00	1.66
Fear	0	0	58.33	0	11.66	15.00
Happiness	0	0	0	71.66	0	0
Sadness	0	0	5.00	8.33	76.66	0
Surprise	0	0	3.33	3.33	1.66	83.33
Under-55 Virtual
Anger	60.00	20.00	0	6.66	3.33	3.33
Disgust	21.66	30.00	3.33	10.00	8.33	1.66
Fear	6.66	1.66	36.66	6.66	10.00	10.00
Happiness	0	38.33	3.33	43.33	8.33	8.33
Sadness	10.00	1.66	1.66	23.33	68.33	28.33
Surprise	1.66	5.00	55.00	10.00	1.66	48.33
Over-55 Photo
Anger	86.66	33.33	13.33	0	0	0
Disgust	13.33	53.33	26.66	6.66	0	0
Fear	0	13.33	53.33	0	20.00	6.66
Happiness	0	0	0	86.66	0	0
Sadness	0	0	0	6.66	80.00	0
Surprise	0	0	0	0	0	93.33
Over-55 Virtual
Anger	46.66	6.66	0	0	0	0
Disgust	13.33	33.33	0	6.66	0	13.33
Fear	13.33	0	40.00	6.66	6.66	20.00
Happiness	0	46.66	0	60.00	0	0
Sadness	26.66	13.33	0	26.66	93.33	13.33
Surprise	0	0	60.00	0	0	53.33

Bold indicates the correctly identified emotion (in %) by that age group and for which media type.

Fig. 9

Mean accuracy between age groups with standard error bars for photographic and virtual eye expressions.

Virtual perception: The real data gathered for the virtual eye regions (see Fig. 10) was further analysed to identify the recognition level of each emotion from within the virtual eye expression renders. The results for each emotion included both age groups and were separated into the appropriate categories of either ‘correct’ or ‘incorrect’. These variables (see Table 3) were then analysed using Fisher's exact tests with a 1 × 2 contingency table. The resulting p-values indicate the proportional significance of the correct vs. incorrect perception of each emotion.

Fig. 10

Breakdown of real data responses for each virtual eye region render.

Table 3

Categorised real data values used in Fisher's exact tests (1 × 2).

Virtual Emotion	Correct	Incorrect	p-value
Anger	43	32	.024* (positive)
Disgust	23	52	.001** (negative)
Fear	28	47	.036* (negative)
Happiness	35	40	>.5
Sadness	55	20	.00001*** (positive)
Surprise	37	38	>.5

A lower p-value indicates a higher significance of positive or negative recognition in the virtual stimuli. Bold is used to show if the recognition level for each virtual emotion is classed as 'correct' or 'incorrect' (i.e. the larger number/score) and that the p-value calculated from those variables is under .05 which indicates a statistical significance.

Fisher's exact tests revealed varying proportional significance across the six virtual stimuli. Sadness (73.3% correct, p < .001) was the best recognised virtual expression followed by anger (57.3% correct, p < .05). Both emotions achieved a correct percentage score of over 50 percent and the resulting p-values show that a notable recognition level was communicated by these two renders. Surprise (49.3% correct, p = 1.0) and happiness (46.6% correct, p = 0.64) attained correct percentage scores close to 50 percent, however, no meaningful amount of perception can be claimed. In contrast, results for fear (37.3% correct, p < .01) and disgust (30.6% correct, p < .01) produced correct percentage scores of under forty percent and therefore generated p-values which suggest a significant negative level of recognition was communicated by these renders. These results are in keeping with the collective dataset which shows fear and disgust to be the least accurately identified emotions across both photographic and virtual eye regions (see Fig. 8). Conceivably the lower recognition percentages for these two emotions could be explained by a number of factors. Firstly, both disgust and fear share similar visual cues to the emotions they were mistaken for by the largest number of respondents, e.g., disgust/happiness display narrow eyelids with pushed up cheeks and fear/surprise display widened eyelids with raised eyebrows. The largest percentage of respondents from both age groups identified the disgust virtual eye region as happiness (see Table 2). This could be explained by findings from previous studies which indicate that the major facial cues for disgust and happiness have been found to reside in the mouth area [32]. Therefore the absence of the leading visual signals may have caused respondent confusion. Disgust is also an emotion which has been shown in previous studies to be frequently not recognised within virtual faces [33] This has been attributed to the lack of wrinkling at the nose base. The FACS indicates this as the main identifying feature of disgust within the eye area. A degree of wrinkling was achieved on the virtual eye region morph, however, the effect may not have been prominent enough (see Fig. 11).

Fig. 11

(a) Micro expressions for disgust identified by the FACS and illustrated by the subject (b) Disgust eye region render.

The largest percentage of respondents from both age groups (see Table 2) identified the fear virtual eye region as surprise. A degree of this error within the data could be attributed to cultural influences which can lead to misinterpretation of expressive facial signals. For example, previous research had indicated that Asians commonly confuse the facial expression of fear for surprise [34]. The aspect of race was intentionally left out of the data acquisition to reduce any potential racial bias. The virtual eye morph for fear perhaps could have been improved by opening the upper eyelids further to expose more sclera as this has been shown to increase the visual impact of the fear expression to a viewer [35].

Summary: In this experiment, we evaluated the perception of eye expressions in photographic and virtual eye regions. Our results showed that the photographic stimuli communicated a greater recognition rate across all emotions than their virtual equivalent. The emotion of sadness attained the highest accuracy percentage for the virtual stimuli and the closest recognition rate between both media types. Further, we found a clear relationship between both data sets showing a prominent decline in recognition for the emotions of disgust and fear (Fig. 8). Although our preliminary results between age group accuracy proved to be not statistically significant further studies with larger sample sizes could provide sufficient evidence which contradicts claims to previous research which suggests that age impairs the ability to identify emotions from natural expressions [36, 37] and virtual expressions [20]. Furthermore, we find that the virtual eye expression renders of sadness and anger generated from our model conveyed a significant level of emotional recognition to respondents. Surprise and happiness indicated no statistically meaningful proportional differences in perception. Finally, fear and disgust displayed an adverse level of perception however these scores could be improved with a higher fidelity of the main subtle cues that the audience uses to identify the emotions and conducting a more comprehensive survey which takes into account respondent's racial identity.

Conclusion

Unlike other studies on the human eyes and emotion, our research is both psychological and CG orientated. The goal of the work was to develop a realistic virtual model of the eye region which can display the micro-expressions necessary to transmit an emotional state to an observer. To attain this goal we combined the use of high-resolution photography, laser scanning and traditional 3D modelling techniques to develop an efficient pipeline capable of producing numerous eye expressions morphs from ground-truth geometry. One key success was that the virtual eye expressions of sadness and anger produced a significant level of emotional perception to respondents. This suggests that potentially CG could be an effective means of research to study eye expressions within the perceptual community. The results point to some areas in which our digital model could be improved in order for the virtual stimuli to be considered “perceptually equivalent” to the real-world photos. Nevertheless, we suggest that for CG artists to achieve better perception results within the eyes of virtual avatars they must directly target the main subtle cues that their audience uses to identify each emotion. Another key finding was the clear constant relationship which can be seen between both photographic and virtual data sets. This suggests that although the medium is influencing the recognition level overall, it does not affect how some emotions appear consistently more difficult to identify than others. For example, disgust and fear show a significantly lower level of recognition across both mediums whereas anger, happiness and sadness attain much higher results.

Limitations of this investigation result from the female gender bias of the visual stimuli and the use of only a single subject for the virtual stimuli creation. The study could be improved upon by using visual stimuli of both sexes and multiple subjects for the virtual stimuli generation. This would allow for comparison between genders and identification of perceptual patterns across emotions which in-turn would produce more objective results. Furthermore, using an online survey did not allow us to control the viewing distance or display equipment used in the experiment. Comparable work [38] has previously used similar methods however this element could have influenced some respondent's judgements. Both the accuracy rate p-values and standard deviation results could be potentially improved by increasing the sample size of the survey the results and analysis in this instance should only be interpreted as a basis for further research, not a reliable conclusion.

In considering directions for future research, it is important to fully investigate the relationship between a person's age and their ability to perceive emotion as this could potentially be a significant area of study.

Our work has provided an initial set of results which are promising and provide a great starting point of utilising CG to investigate the emotional information conveyed by the human eye region. We hope this paper will encourage the CG and psychological communities to conduct further research aimed at understanding the visual cues displayed in human eyes and how these micro-expressions could be realised virtually and accurately to improve our perception of digital humans and increase their acceptance into society.

Declarations

Author contribution statement

S. Barrett, F. Weimer, J. Cosmas: Conceived and designed the experiments; Performed the experiments; Analyzed and interpreted the data; Contributed reagents, materials, analysis tools or data; Wrote the paper.

Funding statement

This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors.

Competing interest statement

The authors declare no conflict of interest.

Additional information

No additional information is available for this paper.