A pilot study of an automated personal identification process: Applying machine learning to panoramic radiographs.

PURPOSE: This study aimed to assess the usefulness of machine learning and automation techniques to match pairs of panoramic radiographs for personal identification.
MATERIALS AND METHODS: Two hundred panoramic radiographs from 100 patients (50 males and 50 females) were randomly selected from a private radiological service database. Initially, 14 linear and angular measurements of the radiographs were made by an expert. Eight ratio indices derived from the original measurements were applied to a statistical algorithm to match radiographs from the same patients, simulating a semi-automated personal identification process. Subsequently, measurements were automatically generated using a deep neural network for image recognition, simulating a fully automated personal identification process.
RESULTS: Approximately 85% of the radiographs were correctly matched by the automated personal identification process. In a limited number of cases, the image recognition algorithm identified 2 potential matches for the same individual. No statistically significant differences were found between measurements performed by the expert on panoramic radiographs from the same patients.
CONCLUSION: Personal identification might be performed with the aid of image recognition algorithms and machine learning techniques. This approach will likely facilitate the complex task of personal identification by performing an initial screening of radiographs and matching ante-mortem and post-mortem images from the same individuals.

Introduction

Mass disaster victim identification is a complex, multidisciplinary task to which forensic dentistry contributes substantially.1 The usefulness of dental records for identifying victims of large-scale disasters is well-known.2 Essentially, personal identification relies on the comparison of ante-mortem (AM) and post-mortem (PM) data, which are often available as dental records from those 2 time points.3

AM dental records can be of various types, including dental charts, dental imaging, and others.4 Dental radiography is a particularly useful and common diagnostic tool in dental practice, which provides highly objective measurements compared to other dental records.5 The diversity of potential applications of dental records to personal identification has been extensively studied and represents a highly effective technique, with a reported accuracy above 98%.6 A variation of the main technique is to perform the estimation not with dental charts, but rather by using AM and PM dental radiographs or computed tomographic (CT) scans with equally good accuracy levels.78 Studies show that panoramic radiographs are particularly useful for PM identification purposes due to the large amount of anatomical information depicted.91011 However, even when AM information is available for comparison, the identification process in the course of a mass disaster is generally complex and time-consuming.

Artificial intelligence (AI), a field of engineering science that trains the ability of computers to perform problem-solving and decision-making tasks automatically, may represent a major advancement in forensic identification. State-of-the-art AI is based on data-driven approaches such as machine learning and deep learning models, which have yielded superb results for image analysis.12 Big data, machine learning, and data mining techniques have been used for predictive studies and forensic automation, representing a novel field of study.1314 The automation of the victim identification process enables its application in mass disasters by computationally comparing anatomical data from AM and PM exams.151617 Several techniques have been described in the literature for the purpose of forensic identification; among the most frequently used are the analysis of dental conditions (such as the assessment of decayed, missing or filled teeth), the number of teeth present, and reference points in anatomical structures.18

This study aimed to assess the usefulness of machine learning and automation techniques to match pairs of panoramic radiographs for personal identification.

Materials and Methods

Ethical approval was obtained from the Research Ethics Committee of the University of São Paulo Dental School (protocol number 79354517000000075). Radiographs were coded and de-identified to protect participants' identities. A pilot study was conducted using panoramic radiographs to assess the applicability of an automated personal identification process.

Two hundred panoramic radiographs from 100 participants (50 males and 50 females) were randomly selected from a private radiological service database. Two radiographs from different time points were obtained for each participant. These radiographs were used to simulate a personal identification process (the most recent panoramic radiograph of each participant was considered the PM record, whereas the older exam was considered the AM record). The shortest time-lapse between the first and the second imaging exams was 9 months. The radiographs were not treated or altered in any way.

Sample size calculation was performed considering a significance level of 5% (α=0.05), 80% test power (β=0.20), mean differences between groups=3, standard deviation=5, and ratio between groups=1/1, resulting in a minimum sample of 200 panoramic radiographs (100 for AM simulation and 100 for PM simulation).

Two examiners were trained and calibrated to perform 14 anatomical measurements (Fig. 1). Both examiners are specialists in forensic dentistry, with experience with human identification and forensic research. For the purpose of training and calibration, examiners made the measurements on 20 panoramic radiographs (10 males and 10 females) using ImageJ version 1.51 (NIH, Bethesda, MD, USA).19 Measurements were obtained in pixels. The intra-class coefficient (ICC), the coefficient of variation (CV), and the Bland-Altman statistics were calculated to verify inter and intra-examiner reliability.

Fig. 1

Marked linear and angular measurements on panoramic radiographs. 1: mandibular angle (right), 2: mandibular angle (left), 3: condyle - coronoid process (right), 4: condyle - coronoid process (left), 5: condyle - condyle, 6: gonion - gonion, 7: coronoid process - gonion (right), 8: coronoid process - gonion (left), 9: mental foramen - mental foramen, 10: mental foramen - sagittal line (right); 11: mental foramen - sagittal line (left), 12: Mental foramen - base of mandible (right), 13: Mental foramen - base of mandible (left).

Initially, 14 linear and angular measurements (Fig. 1) of the radiographs were made by an expert (A.G.O) who had been previously trained and calibrated. Measurements were obtained in pixels using ImageJ version 1.51 (NIH, Bethesda, MD, USA). The following anatomical points were considered for the measurements: the right and left condyles, the base of the mandible, coronoid processes, mental foramen, gonion, mandibular angle, and sagittal line. Eight ratio indices (r1 to r8) were derived from the original measurements and were applied in a statistical algorithm to match radiographs from the same patients, simulating a semi-automated personal identification process (Table 1). A pair of radiographs was considered a match if they presented equal values for at least 5 ratio indices (differences below 0.010). The minimum number of 5 ratio indices was set to balance sensitivity and specificity (for instance, setting a higher number of indices would increase specificity and reduce sensitivity, resulting in a high number of non-matched radiographs).

Table 1

Description of the anatomical points used to perform linear and angular measurements of the panoramic radiographs

Subsequently, a fully automated personal identification process was performed using deep convolutional networks for image recognition in Inception v3. The Inception v3 is an image recognition model trained on ImageNet, a dataset containing over 15 million high-resolution images.20 The model performs the most state-of-the-art computer vision solutions for visual recognition tasks. In this step, the anatomical measurements obtained by the expert were not used since the convolutional networks are able to understand, learn, and distinguish essential visual features on the radiographs. Figure 2 shows the machine learning analysis flowchart. Data were analyzed in R and Python. The following steps were applied: equalization, import, view, analysis, distance measurements, groupings, and finally, pair presentation.

Fig. 2

Flowchart of machine learning analysis.

Adherence of the data to the normality distribution was tested using the Shapiro-Francia statistical test. All variables showed a non-normal distribution. The Mann-Whitney U test was used to assess statistically significant differences between the anatomical measurements obtained by the expert from AM and PM radiographs. The accuracy of the automated personal identification process was calculated as the proportion of radiographs correctly matched. Analyzes were conducted using STATA version 15.1 (StataCorp, College Station, TX, USA) with a significance level of 5%.

Results

The calibration process was considered adequate, with the intra-examiner and inter-examiner ICC above 0.90 for all measurements and the CV below 5%. In the Bland-Altman analysis, the measurements were within the set confidence intervals.

After analysis of the ratio indices by the statistical algorithm, 85% of the AM radiographs had measurements compatible with the corresponding PM radiographs. The same accuracy level was found when the fully automated personal identification process was performed using deep convolutional networks (85%). This method presented advantages over the semi-automated process, such as rapidity, automation, and simplification of the working process.

Examples of matched radiographs performed by the convolutional networks are presented in Figure 3. A successful case of a matching pair of radiographs from the same participant is depicted in Figures 3A and B, demonstrating the ability of the technique to determine qualitative points of similarity between 2 radiographs. Figures 3C-E present a case in which the automation technique identified 2 AM panoramic radiographs (D and E) to compare with the simulated PM radiograph (C) for personal identification. In this case, the correct match can be easily inferred by visual assessment by an expert (D).

Fig. 3

Examples of automatically generated ante-mortem and post-mortem panoramic radiographs.

Figure 4A shows the multidimensional positioning, projection of dimension points, and attempts to adjust distances between points in order to pair similar radiographs. The process of radiograph matching automatically performed by the Machine Learning technique is depicted in supplementary files (Figs. 4B and C).

Fig. 4

Automatically generated ante-mortem and post-mortem matching panoramic radiographs. Pairs of connected circles represent the panoramic radiographs matched by the algorithm.

The adherence of variables to a normal distribution was verified. Only measures 1 and 2 (m1, m2), and ratio indices 1, 2, and 3 (r1, r2, r3) adhered to a normal curve. All variables were statistically treated as nonparametric. Differences between the AM and PM measurements were non-statistically significant for all variables (Table 2).

Table 2

The values of linear and angular measurements of panoramic radiographs at the ante-mortem and post-mortem moments

SD: standard deviation; ns: not significant

Discussion

This pilot study was intended to contribute to the screening of AM and PM images in large-scale accidents with multiple victims and, ultimately, to facilitate the identification process. However, this approach alone does not provide enough evidence to conduct personal identification, which should be carried out according to the recommendations of the International Organization for Forensic Odonto-Stomatology.2122 Although the method described in this study aims to facilitate personal identification, it is important to note that uniqueness in forensic identification is a difficult trait to achieve and may require additional methods.23

The use of automated techniques in forensic dentistry has been applied to several purposes including sex differentiation, age estimation, prediction of mandibular morphology, and identification of bite marks.24 Matsuda et al. (2020) reported the utility of using neural network methods to match a small sample of simulated AM and PM panoramic radiographs. The findings of this study are consistent with the accuracy levels reported by those authors, which ranged from 80% to 100%.25 Similarly, Heinrich et al. (2018) observed a high rate of successful identification (85%) in an automated comparison of AM and PM-simulated panoramic radiographs from a large database.10

In addition to panoramic radiographs, other imaging modalities have been used to assess the applicability of AI to the personal identification process. For instance, the use of automated techniques to analyze cephalograms showed a high ability to predict the mandibular morphology.26 Future studies may investigate the performance of AI in personal identification using computed tomography scans.

In the Bland-Altman analyzes, the measurements were within the confidence intervals set according to the method suggested by the authors, suggesting good intra- and inter-examiner reproducibility.27 No statistically significant differences were found between the AM and PM images according to the measurements in the panoramic radiographs used, and 85% of the AM radiographs presented compatibility with the corresponding PM radiographs.

Although the results of the semi-automated method were satisfactory, performing measurements on radiographs for the purpose of personal identification is time-consuming and may require technical training. To facilitate this process, image recognition algorithms and machine learning techniques were used to perform the initial screening and matching of pairs of radiographs. The findings indicate that using image recognition algorithms for personal identification is as effective as employing manual measurements performed by experts, with the advantage of being time-saving. Other studies have analyzed the accuracy of automatic face recognition, revealing good accuracy.28293031

These findings present important practical implications for personal identification. The machine learning method showed excellent results for matching pairs of simulated AM and PM radiographs, indicating its potential applicability as a screening method in mass accidents. The PM panoramic radiographic technique is a standard forensic procedure that can be executed with appropriate positioning aids in partially or completely edentulous skulls, jawbones, or even fragments of jawbones. The technique has been described elsewhere.32 However, a combination of other expert methods is usually required to determine positive identification.3334

The diverse possibilities offered by dental records have been extensively studied for personal identification, and dental records can be used for positive identification.6 Fujimoto et al. (2016)35 proposed a new method of personal identification, the IDOL method, which uses reference points in anatomical structures on AM panoramic radiographs compared to PM computed tomography. Satisfactory results were obtained without relying on dental treatments as a reference. Although dental measurements are hallmarks for PM identification, this analysis was primarily based on anatomical points obtained from facial bone structures. This may be considered a strength of the study since it enables the applicability of the method with edentulous individuals and other contexts in which teeth may be absent from corpses.

The use of machine learning in the forensic field is promising and opens a new field of research to contribute to the expert practice of forensic dentistry. An important limitation of this study is that the AM and PM conditions were simulated. Future studies may use actual AM and PM data to investigate the applicability of using machine learning algorithms to aid in mass disaster victim identification processes.

This pilot study indicates that personal identification can be performed with the aid of image recognition algorithms and machine learning techniques, which may facilitate the expert forensic workflow by performing an initial screening and matching of AM and PM images. These tools were shown to be reliable and time-saving, and do not require experts to perform manual measurements on radiographs.