Changes in a facial recognition algorithm following different types of orthognathic surgery: a comparative study.

Objectives: Contemporary biometric technologies have been gaining traction in both public and private security sectors. Facial recognition is the most commonly used biometric technology for this purpose. We aimed to evaluate the ability of a publicly available facial recognition application program interface to calculate similarity scores of presurgical and postsurgical photographs of patients who had orthognathic surgery. Materials and
Methods: Presurgical and postsurgical photographs of 75 patients who had orthognathic surgery between January 2018 and November 2020 in our department were used. Frontal photographs of patients in relaxed and smiling states were taken. The patients were classified into three groups: Group 2 had one-jaw surgery, Group 3 had two-jaw surgery to correct mandibular prognathism, and Group 4 had two-jaw surgery to correct facial asymmetry. For comparison, photographs of 10 participants were used as controls (Group 1). Two facial recognition application programs (Face X and Azure) were used to assess similarity scores.
Results: The similarity scores in the two programs showed significant results. The similarity score of the control group, which did not undergo orthognathic surgery, was the highest. The results for Group 2, Group 3, and Group 4 were higher in the order of Group 2, Group 3, and Group 4.
Conclusion: In this study, all orthodontic patients were recognized as the same person using the face recognition program before and after surgery. A significant difference in similarity results was obtained between the groups with both Face X and Azure and in both relaxed and smiling states.

I. Introduction

A biometric system is an automated technology that assesses an individual based on physiological or biological characteristics to confirm their identity1,2. Examples of physical characteristics used in biometric technologies include fingerprints, iris patterns, retina shape, hand shape, facial morphology, and vein patterns3,4. The face is the most common and familiar biometric element in daily life. Facial recognition can be used for various purposes in the real world such as surveillance, access control, and identity verification5. However, the human face is not ideal compared with other biometric factors. In general, facial data are less accurate because they are heavily influenced factors such as cosmetics, disguise, and lighting6. However, biometric recognition using faces can be easily implemented without active engagement or cooperation by the individuals.

Facial recognition technology compares new images with images in a searchable database, and identifies matches and potential matches7. Facial recognition is primarily represented by two processes: authentication and identification. The authentication process can be defined as a one-to-one match that associates a facial image with a bank of available face image data with a matching personality. Facial identification is a one-to-many problem of matching the query image of a face with the available images in the face database8.

Orthognathic surgery improves facial proportions and harmony by altering the positions of the upper and lower jaws9 by changing the hard tissue and soft tissue10. Orthognathic surgery is classified as single jaw surgery, which deforms only the mandible, and double jaw surgery, which deforms both the upper and lower jaws. There are various types of double jaw surgery, such as canting, yawing, advancement, and posterior impaction, depending on the surgical method for the maxilla. We aimed to evaluate a publicly available facial recognition application program interface for calculating similarity scores for presurgical and postsurgical photographs of patients undergoing orthognathic surgeries.

II. Materials and Methods

1. Subjects

This study included patients who had orthognathic surgery at the Department of Oral and Maxillofacial Surgery, Dankook University Dental Hospital between January 2018 and November 2020. Patients were excluded if they had orthognathic surgery after November 2020, did not have a follow-up picture 1 year after surgery, had plastic surgery in addition to orthognathic surgery, or if their facial pictures were of poor quality. A total of 75 patients were divided into four groups: one control group and three groups corresponding to the experimental groups. The experimental groups comprised patients that had one-jaw surgery, patients that had two-jaw surgery for mandibular protrusion, and patients that had two-jaw surgery for facial asymmetry.(Fig. 1)

Fig. 1

Example facial photographs of each group: (A) Group 1 patient, (B) Group 2 patient, (C) Group 3 patient, and (D) Group 4 patient. (Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry)

This study was approved by the Institutional Review Board of Dankook University Dental Hospital (IRB No. DKUDH IRB 2022-3-008). Written informed consent was obtained from all patients.

2. Methods

The surgery was performed by an oral and maxillofacial surgeon, and sagittal split ramus osteotomy was performed on all mandibles. In the case of the maxilla, maxillary canting was not performed in patients of Group 3; however, maxillary canting correction was performed in Group 4. Following the surgery, the patients collectively underwent intermaxillary fixation. Following their discharge, a periodic follow-up was performed, and clinical and radiographic images were acquired.

3. Study methods

The study was conducted using photos of expressionless and smiling expressions of 75 patients taken prior to orthognathic surgery and 1 year following surgery. For this study, two facial recognition programs were used, Face X and Azure, and the face comparison function of each program was employed. Facial similarity was obtained by applying the facial photographs of all patients taken before surgery and 1 year after surgery to the program, and the results were analyzed.

4. Analysis

Statistical analyses were conducted by a single researcher using IBM SPSS Statistics (ver. 28; IBM, Armonk, NY, USA), and a one-way ANOVA was performed. Facial similarity values were compared between each group; P<0.05 were considered to indicate statistically significant differences.

III. Results

1. Study model characteristics

Group 1, the control group, comprised eight male and two female participants, with a mean age of 28.90 years. Group 2, i.e., the group that had one-jaw surgery, comprised 21 individuals (14 male and 7 female), with a mean age of 22.09 years. Group 3, the group that had two-jaw surgery for mandibular protrusion, comprised 28 individuals (13 male and 15 female), with a mean age of 22.48 years. Group 4, the group that had two-jaw surgery for facial asymmetry, comprised 26 individuals (10 male and 16 female), with a mean age of 22.46 years.(Table 1)

Table 1

Sample characteristics

	Group 1	Group 2	Group 3	Group 4
Sex
Male	8	14	13	10
Female	2	7	15	16
Mean age (yr)	28.90	22.09	22.48	22.46

Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry.

2. Facial similarity results (Face X)

Following comparison of the relaxed facial photographs, the similarity values before and after surgery were 0.86±0.15 in Group 1, 0.79±0.07 in Group 2, 0.76±0.10 in Group 3, and 0.74±0.10 in Group 4. The mean value was 0.78±0.09. The P-value was 0.009.(Tables 2, 3)

Table 2

P-value obtained from analysis on each program

	P-value
Face X (relaxed state)	0.009
Face X (smiling state)	0.005
Azure (relaxed state)	<0.001
Azure (smiling state)	<0.001

In both programs, the P-value was lower than 0.001, indicating a significant result.

Table 3

Similarity score (Face X – relaxed state)

	Mean±SD	95% CI
Face X (relaxed state)
Group 1	0.86±0.15	0.84-0.88
Group 2	0.79±0.07	0.72-0.80
Group 3	0.76±0.10	0.73-0.80
Group 4	0.74±0.10	0.70-0.78
Mean	0.78±0.09	0.75-0.79

(SD: standard deviation, CI: confidence interval) Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry.

Mean similarity score was highest in the order of Group 1, Group 2, Group 3, and Group 4.

Following comparison of the smiling state photographs, the preoperative and postoperative similarity values were 0.85±0.16 in Group 1, 0.78±0.08 in Group 2, 0.76±0.09 in Group 3, and 0.75±0.07 in Group 4. The mean value was 0.78±0.08. The P-value was 0.005.(Tables 2, 4)

Table 4

Similarity score (Face X – smiling state)

	Mean±SD	95% CI
Face X (smiling state)
Group 1	0.85±0.16	0.84-0.87
Group 2	0.78±0.08	0.75-0.82
Group 3	0.76±0.09	0.73-0.80
Group 4	0.75±0.07	0.72-0.78
Mean	0.78±0.08	0.76-0.79

(SD: standard deviation, CI: confidence interval)

Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry.

Mean similarity score was highest in the order of Group 1, Group 2, Group 3, and Group 4.

3. Facial similarity results (Azure)

Following comparison of the relaxed state photographs, the similarity values before and after surgery were 0.98±0.01 in Group 1, 0.96±0.01 in Group 2, 0.94±0.02 in Group 3, and 0.92±0.01 in Group 4. The mean value was 0.95±0.02. The P-value was less than 0.001.(Tables 2, 5)

Table 5

Similarity score (Azure – relaxed state)

	Mean±SD	95% CI
Azure (relaxed state)
Group 1	0.98±0.01	0.97-0.99
Group 2	0.96±0.01	0.95-0.96
Group 3	0.94±0.02	0.94-0.95
Group 4	0.92±0.01	0.92-0.93
Mean	0.95±0.02	0.94-0.95

(SD: standard deviation, CI: confidence interval)

Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry.

Mean similarity score was highest in the order of Group 1, Group 2, Group 3, and Group 4.

Following comparison of the smiling state photographs, the similarity values before and after surgery were 0.98±0.01 in Group 1, 0.96±0.01 in Group 2, 0.94±0.02 in Group 3, and 0.93±0.01 in Group 4. The mean value was 0.95±0.02. The P-value was less than 0.001.(Tables 2, 6)

Table 6

Similarity score (Azure – smiling state)

	Mean±SD	95% CI
Azure (smiling state)
Group 1	0.98±0.01	0.98-0.99
Group 2	0.96±0.01	0.95-0.96
Group 3	0.94±0.02	0.94-0.95
Group 4	0.93±0.01	0.93-0.94
Mean	0.95±0.02	0.94-0.95

(SD: standard deviation, CI: confidence interval)

Group 1: the control group, Group 2: the group that had one-jaw surgery, Group 3: the group that had two-jaw surgery for mandibular protrusion, Group 4: the group that had two-jaw surgery for facial asymmetry.

Mean similarity score was highest in the order of Group 1, Group 2, Group 3, and Group 4.

IV. Discussion

In this study, facial comparison was performed by extracting features from the probe images and comparing the geometrical relationship between the constructed points with the existing database to identify the closest match and record the similarity as a percentage11. This study was performed assuming that the soft tissue changes after surgery would affect the facial similarity. Frontal photos were used, and the maxillary canting correction was considered to induce a greater change in the face; as a result, the groups were classified accordingly.

Two facial recognition programs (Face X and Azure) were used to analyze facial similarity. There was a slight difference in the overall similarity values between the two programs. When the same photograph was analyzed, the facial similarity value obtained using Face X was less than that obtained using Azure. To identify the reason for this difference, we assessed the analysis information from the two programs. Face X conducted similarity analysis using a total of 68 landmarks (Fig. 2), and Azure conducted an analysis using 27 landmarks.(Fig. 3) This difference in the number of contributes to the overall similarity values. Specifically, the similarity value is low because more changes were recognized when analyzing more landmarks. When assessing the similarity of the numerical results, the results of both the Face X and Azure programs yielded high results in the following order: Group 1, Group 2, Group 3, and Group 4.

Fig. 2

Landmarks analyzed by the Face X program.

Fig. 3

Landmarks analyzed by the Azure program.

There were differences in the analysis methods and results for each program; however, there were no differences in pre- and postoperative photos of the same person when comparing facial expressions. Therefore, according to this study, orthognathic surgery did not affect a person being recognized. This may be attributed to the change in soft tissue morphologies in relation to the position of the maxilla and mandible after orthognathic surgery; however, because the measurement points analyzed by the face recognition program are distributed across the entire face, there were few special changes in the measurement points.

According to a study by Dragon et al.7 on the relationship between facial recognition technology and orthognathic surgery, the mean similarity value was 97 (in quadrant 96-99) for patients who had two-jaw surgery and 99 (in quadrant 99-99) for patients who had one-jaw surgery. This study confirmed that the similarity level was lower in the case of two-jaw surgery than in the case of one-jaw surgery.

This study had some limitations. First, a person’s face is three-dimensional; however, in this study, facial comparison was performed using the patient’s clinical photographs; therefore, the three-dimensional element of the actual face was not considered. Second, photographs of the patient’s face before and 1 year after surgery were not taken by the same photographer. Third, the location and environment where the facial photo was taken were the same; however, small differences in the brightness and shooting angle of the photo were impossible to eliminate. Finally, the number of patients in each group was different. When conducting additional studies, a larger sample size is necessary. Furthermore, more meaningful results will be obtained if a study is conducted to recognize and analyze the three-dimensional faces of an actual patient in real time.

V. Conclusion

In this study, facial similarity was analyzed by applying before and after facial photographs of patients who had orthognathic surgery to facial recognition programs. The Face X and Azure programs were able to identify people who have undergone orthognathic surgery. When considering the similarity results, a significant difference was observed between the groups using both Face X and Azure and in both the relaxed and smiling state photographs. Strong similarity results were observed in the group that had two-jaw surgery for mandibular protrusion and the group that had two-jaw surgery for facial asymmetry.