Can a spontaneous smile invalidate facial identification by photo-anthropometry?

PURPOSE: Using images in the facial image comparison process poses a challenge for forensic experts due to limitations such as the presence of facial expressions. The aims of this study were to analyze how morphometric changes in the face during a spontaneous smile influence the facial image comparison process and to evaluate the reproducibility of measurements obtained by digital stereophotogrammetry in these situations.
MATERIALS AND METHODS: Three examiners used digital stereophotogrammetry to obtain 3-dimensional images of the faces of 10 female participants (aged between 23 and 45 years). Photographs of the participants' faces were captured with their faces at rest (group 1) and with a spontaneous smile (group 2), resulting in a total of 60 3-dimensional images. The digital stereophotogrammetry device obtained the images with a 3.5-ms capture time, which prevented undesirable movements of the participants. Linear measurements between facial landmarks were made, in units of millimeters, and the data were subjected to multivariate and univariate statistical analyses using Pirouette® version 4.5 (InfoMetrix Inc., Woodinville, WA, USA) and Microsoft Excel® (Microsoft Corp., Redmond, WA, USA), respectively.
RESULTS: The measurements that most strongly influenced the separation of the groups were related to the labial/buccal region. In general, the data showed low standard deviations, which differed by less than 10% from the measured mean values, demonstrating that the digital stereophotogrammetry technique was reproducible.
CONCLUSION: The impact of spontaneous smiles on the facial image comparison process should be considered, and digital stereophotogrammetry provided good reproducibility.

Introduction

The spread of surveillance cameras and smartphones has made the use of images during crime scene analysis a common practice among forensic experts.123 This practice has helped not only to identify missing or unknown people,24 but also to estimate age in suspected cases of child sexual exploitation.5 It is therefore necessary to develop and test methods that can be applied in these situations. In this context, analysis of the human body for forensic purposes comprises a field of study that has challenged experts from different areas.267

In a criminal investigation, images may constitute the only available means of probative analysis578 when it is crucial to identify people by images obtained from surveillance cameras.910 Images are an important means of recording information and have already been used to pass sentences and as proof of materiality.10

However, some factors make the use of morphological analysis and overlapping methods, such as facial image comparison, difficult.2111213 These limitations include camera quality, image resolution, ambient lighting, and the suspect's movement and spatial orientation in relation to the camera.312131415 In addition, many authors have discussed the possibility that facial expressions can significantly influence the facial image comparison process12161718 and whether these expressions are reproducible,1920 especially under spontaneous conditions, when their reproducibility might limit facial movement analysis.18 For all these reasons, photo-anthropometry has not been recommended as a facial image comparison method.11

Nonetheless, advances in image capture technology have led to the emergence of video cameras that show images from different angles and carry out 3-dimensional (3D) reformatting of people's images.21 In the medical field, digital stereophotogrammetry aids the 3D study of the human face.22232425 In the forensic context, this technique has provided greater security with regard to image analysis and has opened new possibilities in facial identification.2627

Given that morphometric changes of the face during facial expressions remain little-explored and that complete facial analysis based on 3D images obtained under controlled conditions may enable a greater explanation and understanding of the facial identification process, this study aimed to establish which photo-anthropometric measurements change the most during spontaneous smiles and to evaluate the reproducibility of these measurements in 3D images obtained by digital stereophotogrammetry.

Materials and Methods

This cross-sectional study was carried out with approval of the Research Ethics Committee of the General Hospital of the Medical School of Ribeirão Preto of the University of São Paulo (protocol number: 86380818.8.0000.5440). Based on previous studies,1718 the sample size of this pilot study consisted of 10 female participants, aged between 23 and 45 years, who had previously provided consent and had been invited to participate in the research. The selected participants were healthy and had no history of trauma or craniofacial pathologies. In addition to these delimitations, the sample did not include participants who i) were unwilling to have their images taken for personal reasons; ii) had scars; and iii) had face adornments that would make reading the images difficult.

With clean skin and their hair tied back with the aid of a hair tie, each participant was evaluated by 3 different examiners, who had previously been calibrated to obtain 3D images by digital stereophotogrammetry. The image acquisition process was conducted in triplicate and started with the marking of 39 anthropometric points that were identified through a visual inspection of the participants' faces. Black eyeliner (Quem Disse, Berenice?®, Interbelle Comércio de Produtos de Beleza Ltda, São Paulo, SP, Brazil) was used to tag the points. Then, the participant was seated in front of a Vectra M3® system device (Canfield Scientific Inc., Fairfield, NJ, USA) with occluded teeth, closed lips, and a relaxed face, to reproduce a resting face. The obtained image corresponded to group 1 (resting face, Fig. 1). Immediately after the resting face image was captured, the participant remained in the same position and was induced to smile spontaneously, and the captured image was allocated to group 2 (spontaneous smiling face, Fig. 1). The digital stereophotogrammetry device obtained the images with a 3.5-ms capture time, which prevented undesirable movements of the participants. Next, the participant had the markings removed with the help of a make-up remover (Quem Disse, Berenice?®, Interbelle Comércio de Produtos de Beleza LTDA, São Paulo, SP, Brazil).

Fig. 1

Identified landmarks (green points): A. Three-dimensional (3D) photograph, resting face (group 1). B. 3D photograph, face with a spontaneous smile (group 2). The abbreviations in blue refer to the names of landmarks, and the full descriptions of each landmark can be found in Tables 1 and 2.

In the 3D images, all the landmarks (Tables 1 and 2) were identified by using the software Vectra Analysis Module® (Canfield Scientific Inc., Fairfield, NJ, USA), and 42 linear measurements were made in units of millimeters (Fig. 2). Although Figure 1 shows the marking of the landmarks, the measurements were only taken in the middle and lower facial thirds because previous research1718 raised the hypothesis that these regions may be the most relevant for distinguishing between a resting face and a spontaneous smiling face when linear measurements are made.

Table 1

Description of the identified midline landmarks. Adapted from Ferrario et al.28

Table 2

Description of the identified bilateral landmarks. Adapted from Ferrario et al.28

Fig. 2

Specification of all the linear measurements made in millimeters (red lines). Measurements performed on the right (A), frontal (B), and left (C) views of the participant with a smiling face are illustrated.

The data were subjected to multivariate statistical analysis. To conduct the analyses, Pirouette® version 4.5 (Info-Metrix Inc., Woodinville, WA, USA) was used, and no pretreatment was performed because the data had the same nature and dimension. Two methods, unsupervised and supervised, were employed. Principal component analysis (PCA) was applied as an exploratory evaluation to reduce the dimensionality of the system of variables and to observe possible groupings. Thus, the samples and variables were rewritten in a new system of axes that accounted for variance in the system. These axes are called principal components or factors. The coordinates of the samples through the principal components were presented by graphs called scores, while the variables were presented by graphs referred to as loadings.29 For the supervised approach, the partial least squares for discriminant analysis model (PLS-DA) was applied to verify the adjustment of the samples to classes. Thus, the proposed technique aimed to classify the samples according to the previously defined groups.303132

The results were evaluated based on the parameters Q2, R2, root mean square error of validation (RMSEV), and root mean square error of calibration (RMSEC), as described in Equations 1 to 4. To guarantee the model quality, it must be the case that R2>Q2 and RMSEC

Q2: internal correlation coefficient:

R2: correlation coefficient for calibration:

RMSEV: root mean square error of validation:

RMSEC: root mean square error of calibration:

Finally, the samples were statistically analyzed using a univariate approach with Microsoft Excel® (Microsoft Corp., Redmond, WA, USA) to assess the reproducibility of spontaneous smiles and to test the examiners' calibration. For this purpose, the standard deviation for each measurement in each participant was calculated in order to assess the agreement of the measurements made by each examiner and to observe whether there was subjectivity in the location of these reference points on the face.

Results

Each photographed individual was considered to be a sample, and the distances acquired in the measurement process were used as variables. The sample set was divided into 2 classes (groups 1 and 2). In each of these classes, examiners 1, 2, and 3 were distinguished by the colors black, red, and blue, respectively. In the first PCA, there was an outlier; that is, an anomalous measurement that differed from the other measurements. This was related to the measurements that all the examiners made on participant 9. However, the groups had already been formed in a different way, so this outlier did not prevent the separation of the groups in the observed classes (Fig. 3).

Fig. 3

Principal component analysis scores for the data obtained from the resting (class B, group 1) and spontaneous smiling (class A, group 2) faces in each participant (P). The black, red, and blue samples represent examiners 1, 2, and 3, respectively.

This sample was removed to improve dispersion and to enable a more accurate assessment, which made it possible to distinguish between the 2 classes that were obtained according to the scores shown in Figure 3. Class A represents the measurements made in group 2, and class B represents the measurements performed in group 1. The separation occurred in principal component 2 (factor 2).

Concerning all the information entered for the PCA, 99.76% of it was contained in 2 principal components, which reduced the dimensionality of the system of variables. The loadings (Fig. 4) show the distribution of the measurements that were evaluated in each of the principal components.

Fig. 4

Loadings in the principal component analysis. Class A represents the measurements made in group 2 (spontaneous smiling face), and class B represents the measurements performed in group 1 (resting face).

The samples were separated according to the second principal component, as shown in Figure 3. Figure 4 depicts the variables that were responsible for discriminating the samples. The observed overlap meant that some variables had a similar influence on sample discrimination. Table 3 lists the numerical values for Figure 3; the values were organized in decreasing order of influence in discriminating the samples. For class A, most values for the second principal component were positive, whereas most values for the second principal component were negative for class B. Furthermore, the more negative the value in Table 3, the higher the weight the sample had on belonging to class B. Conversely, the more positive this value, the higher the weight the sample had on belonging to class A. Several measurements (left Go - Pg, right Go - right T, right Go - Pg, Gn - left Go, Gn - right Go, Ls - Sto, right T - left T, and right Zy - left Zy; Tables 1 and 2) had a slight influence because their values were close to 0 for the second principal component. Therefore, these measurements were removed from the original data group, given that they had no significant discriminatory influence. To drive decision-making, it was decided to remove measurements in a cut-off zone between − 0.03 and 0.03.

Table 3

Measurements that influenced the separation of the classes, organized in descending order. Class A represents the measurements made in group 2 (spontaneous smiling face), and class B represents the measurements performed in group 1 (resting face)

The variables that most strongly influenced the separation between classes A and B corresponded to the distances between the reference points of the labial/buccal region and the mandible. This fact confirmed the initial hypothesis that these measurements would be able to differentiate between the assessed classes and would be the most important in separating them. The unsupervised analysis revealed that there was a relative separation with few measurements that could be used to create a model to classify these groups.

After exploring the system by PCA, a second multivariate statistical evaluation was conducted. In this evaluation step, we used PLS-DA as a supervised approach. The overall idea was to evaluate the classes previously observed by PCA. The objective was to assess whether it would be possible to differentiate the resting and spontaneous smiling faces among the different participants.

In the PLS-DA model, the data matrix (X) can be correlated with a vector y, which takes on qualitative values (1 and − 1, for example), and each class is discriminated through these values. Thus, for the PLS-DA training model, class 1 values were assigned to samples referring to the dataset of participants with a spontaneous smiling face and class − 1 values to samples corresponding to the dataset of participants with a resting face. Outliers were not detected.

To assess the robustness of the model created by applying the statistical-mathematical procedure, the parameters Q2, R2, RMSEV, and RMSEC were observed. To ensure model quality, 2 basic conditions for the multivariate regression models (PLS-DA in this case) were used, namely R2>Q2 and RMSEC

Table 4

Parameters obtained for the model created to classify the resting and spontaneous smiling faces of female participants.

Based on the information presented in Table 4, it was decided that the best model involved 3 principal components (factor 3). This choice was made because the total amount of information that was accumulated between principal components 3 (99.78%) and 4 (99.86%) was small, so the addition of more principal components to the model did not provide significant improvements. However, 2 principal components (factor 2) afforded a less reliable model. Hence, the parameters were: Q2=0.878, R2=0.904, RMSEV=0.349, and RMSEC=0.318.

Figure 5 represents the PLS-DA model that was created by using 3 principal components. Samples with values above 0 were related to the participants with resting faces; samples with values below 0 corresponded to the participants with spontaneous smiling faces. These results showed that the model was robust enough to distinguish between faces given that none of the samples crossed the 0 axis on Pred Cal. Therefore, there was no error in classification.

Fig. 5

Representation of the model developed between the resting (values above zero) and spontaneous smiling faces (values below zero).

The PLS-DA model can distinguish between resting and spontaneous smiling faces, as in the case of the groups in this study. To ensure that the measurements were performed correctly, the reproducibility of this study was assessed. In this approach, it was evaluated whether a spontaneous smile was being made at the time of the photographs. For this purpose, the examiners' accuracy was considered when they made the markings and measurements. A univariate statistic was employed to verify the standard deviation that was related to each measurement for each examiner per sample (Fig. 6)

Fig. 6

Graph correlating the measurement locations (x axis), the evaluated individuals (P) (z axis), and the standard deviation (y axis) for each performed measurement.

Thus, an analysis of Figure 6 revealed certain regions with larger standard deviations (peaks of greater intensity), which corresponded to the places where the measurements were less reproducible and where the examiners' measurements were more discrepant. However, the data generally showed low standard deviations because they differed by less than 10% in relation to the measured mean values. Additionally, the measurements that resulted in the best measurement performance among the examiners were linked to the labial/buccal and mandible region, as follows: right Ch - left Ch, left Chk - left T, right Chk - right T, left Cph - left Ch, right Cph - right Ch, right Cph - left Cph, left Go - left T, right Go - left Go, Li - left Ch, Li - right Ch, Ls - left Cph, Ls - right Cph, Ls - Li, N - Gn, N - Me, N - Pg, Sn - Gn, Sn - Me, Sn - Pg, and Sto - Li (for definitions, see Tables 1 and 2).

Discussion

The presence of facial expressions in images that are used in the facial image comparison process can compromise the facial identification process.12161718 However, their true influence remains unclear. In this context, digital stereophotogrammetry enables the face to be morphometrically studied by means of high-quality and realistic 3D images, which result from reformatting 2-dimensional (2D) photographs that are captured in different angles from similar distances.22243334

Due to the speed with which the photographs are captured and to the absence of physical contact with the analyzed region, biases related to the participants' movement or to tissue changes resulting from physical contact are minimized.223335363738 Gibelli et al.17 used the stereophotogrammetric technique to superimpose 3D images of 10 male participants who displayed different facial expressions (happy, sad, angry, surprised) onto the respective 3D images of their resting faces. By applying root mean square values, the authors found that the happy expression was responsible for considerable changes in the mouth and chin.

Therefore, the facial identification process requires a prior suspect's registration so that comparison is possible.11 Moreover, despite its advantages, the technology that allows 3D facial images to be captured still is not widespread. In this context, 2D photographs are still necessary, and this type of material is more accessible. Nevertheless, Kleinberg et al.39 compared angles and proportionality indices between sets of photographs of different people and concluded that the use of 2D photographs in facial identification may be subject to the presence of confounding variables, such as facial expressions, which can prevent consistent results from being obtained. Kleinberg et al.39 presented suggestions that should be explored, which were included in this research.

Based on the results of Kleinberg et al.39 and Gibelli et al.17, this study aimed to assess which linear measurements performed on 3D images would be particularly relevant for discriminating between a resting face image and the corresponding spontaneous smiling face image. The multivariate statistics applied in this research made it possible to verify that the data were naturally grouped into 2 distinct classes, and that a few linear measurements performed on the 3D images were particularly relevant for separating the evaluated groups (Table 3).

Although Gibelli et al.2 pointed out that facial expressions could cause small changes in the face as a whole, and that facial expressions are important because they affect the facial image comparison methods, here the main facial changes during spontaneous smile were detected in the labial/buccal region. As expected, from a practical viewpoint, the results of the present research suggest that linear measurements between different landmarks of the labial/buccal region should be chosen when deciding whether to apply photo-anthropometric analysis as a method of human identification in these situations. This finding was clear from the application of the PCA technique, which allowed the natural organization of the data to be understood and enabled to verify that some measurements were more influenced by a spontaneous smile, while other measurements were not significant for any of the evaluated cases.

In short, regarding the measurements that had a more marked influence on group separation, some measurements were more influential than others (Table 3). Within class B (group 1), for example, the Ch - Chk measurements (right and left) influenced class separation less than the Ch - T measurements (right and left). Although, because of their negative values, the Ch - Chk measurements (right and left) had a weight on separating class B (group 1) from class A (group 2) too, as can be seen in Table 3. This fact, added to the influence that the right Ch - left Ch and Li - Ch (right and left) measurements had on the separation of class A, demonstrates the importance of cheilion displacement during a spontaneous smile, which can often be observed in a 2D image context.40

The supervised classificatory technique (PLS-DA) reinforced the PCA conclusions. In this study, a model that can be used to predict the participants' spontaneous smiling and resting faces was created. Thus, as shown in Figure 5, the PLS-DA model correctly separated groups 1 and 2, with the spontaneous smiling face samples having greater dispersion in the Pred Cal axis as compared to the resting face samples. This indicated that, in practice, even though a person may exhibit different smile amplitudes spontaneously, small changes in the lip/buccal region can influence the linear measurement values obtained in this region, as observed by Gibelli et al.2 Therefore, during the facial image comparison process, safely delimiting the presence or absence of facial changes related to a spontaneous smile is relevant. However, for other participants to be included, it is necessary to expand the sample and reassess the statistical parameters.

In a 3D context, Velemínská et al.37 explained that significant changes in the facial surface occur in the anterior (middle and sides of the cheek) and posterior (corners of the mouth and areas around it) directions when a 3D image of the smiling face is compared with a 3D image of the resting face. In addition, Velemínská et al.37 emphasized that 3D images allow facial changes to be studied in their entirety without information loss. Therefore, research must be carried out to evaluate the influence of the loss of dimension that occurs in 2D photographs, mainly because Martos et al.41 found promising results when they estimated dimensions and proportionality indices in 3D images formed from 2D photographs.

Nonetheless, the spontaneity of facial expressions must be considered18 because the standardized smile does not reflect the range of movements that the lips can perform, in addition to being subject to inter-individual variability.40 In this context, to test spontaneous smile reproducibility, a univariate statistic was used, which made it possible to verify that certain measurements had smaller standard deviations; in other words, they were more reproducible, while other measurements afforded a greater standard deviation (Fig. 6).

With regard to facial expression reproducibility, Özsoy et al.42 used a portable 3D scanner and found that the evaluated facial expressions (surprised, angry, sad, scared, happy, disgusted) were reproducible within subjects even after a 3-month interval, as well as in intra- and inter-observer analyses, wherein the faces were scanned twice consecutively. Although Özsoy et al.42 did not evaluate smile spontaneity, their approach, as well as the approach of the present research, enabled safe use of the data by demonstrating reproducibility in both the qualitative and quantitative aspects. Reinforcing this idea, Sawyer et al.20 evaluated the reproducibility of 9 facial expressions through digital stereophotogrammetry in different sessions that ranged from 15 minutes to 1 month apart, to find that all the expressions were reproducible after a 15-minute interval. For a 1-month interval, Sawyer et al.20 found statistically significant differences in just 2 facial expressions, which were “smile-with-lipsopen” and “blow-out-the-cheeks”, although Sawyer et al.20 recognized that these differences were clinically insignificant.

Hence, the study of facial expressions in spontaneous conditions is important because these expressions are present in everyday situations in which facial identification may be required. As highlighted by Tarantili et al.43, analyses of facial expressions by photo-anthropometric methods requires caution related to landmark location and facial tissue movement. Methods that associate landmark location with the advantages of digital stereophotogrammetry have proven considerably reliable in terms of measurements.35 As for facial tissue movement during spontaneous smiles, Tarantili et al.43 used a digital video camera to assess the reactions of 15 children while they were watching a funny cartoon video, and found that the width of the mouth (right Ch - left Ch) increased by an average of 27% and the height of the upper lip decreased by an average of 28% (with both measured in relation to the resting face). This demonstrated the changes that these distances undergo in these situations.

Tarantili et al.43 also emphasized that the displacement of points such as the cheilion can differ among people; that is, the smile can present different degrees of asymmetry. They also highlighted that 3D image capture methods employing landmark location and inducing spontaneous smiles by verbal stimuli can create an artificial environment that can interfere with the spontaneity condition, which may have been a limitation of this research. Notwithstanding, this research is important because it has demonstrated that measurements related to the labial/buccal region can be reproducible (Fig. 6) although they are conditioned to subjective factors intrinsic to each participant, such as inter-individual variability and the greater or lesser range of the smile when facing each examiner.40

Thus, the results of this study may support new research that uses univariate and multivariate analyses to assess which linear measurements should be considered in other facial expressions for the purpose of facial identification because the microexpressions originating with the smile did not influence the separation of the evaluated groups. It is possible that application of the proposed methodology to 2D photographs can direct the facial identification process in places where techniques for obtaining 3D images are not yet present. Additionally, as digital stereophotogrammetry devices for facial analysis have been widely used in the medical field,22232425 large image databases from different populations around the world can be used to test the methodology proposed in this research.

Although PLS-DA or other regression methods can be used for prediction, multivariate methods are helpful to understand systems with high dimensionality. These methods are not just about predictions. Multivariate tools can be applied to explore differences in data sets with many variables to obtain information that would not be easily observed because of the dimensionality. Simple regression is not enough to understand a system with many variables. In the specific case of PLS-DA, the use of class associations makes it possible to explore the system more clearly, indicating parameters to evaluate the quality of the classification.4445 Multivariate regression methods are used to deal with multivariate systems, analogously to the use of univariate regression to understand univariate data.46 Based on this information, it is possible to comprehend how the system behaves. This knowledge can eventually be used for forecasting, but it is not mandatory. In the specific case of this work, a use that has not yet been explored in the areas of dentistry and dental imaging was proposed. The multivariate approach was shown to be useful to recognize differences between the resting and spontaneous smiling faces.

In conclusion, a spontaneous smile may cause morphometric changes in the face, so its influence should be considered when facial identification is necessary, especially if photo-anthropometric analyses are applied to the labial/buccal region. Indeed, measurements such as right Ch - left Ch, Li - Ch (right and left), and Sto - Li undergo major changes during spontaneous smiles. As demonstrated by the PLS-DA results, such changes enabled a safe distinction between the resting and spontaneous smiling faces, which can be particularly useful in a forensic context of assessment. Thus, this study can help to create more reliable methods for face distinction by developing a database and improving the options that are available for these purposes, including visual comparison. However, as shown in PCA, some measurements did not undergo significant changes, which suggested that microexpressions in certain regions of the face may not invalidate the facial identification process. In addition, measurements related to the smile were shown to be reproducible, but photo-anthropometric analyses should be performed with caution because some measurements were more reproducible than others.