Deep learning-based facial image analysis in medical research: a systematic review protocol.

INTRODUCTION: Deep learning techniques are gaining momentum in medical research. Evidence shows that deep learning has advantages over humans in image identification and classification, such as facial image analysis in detecting people's medical conditions. While positive findings are available, little is known about the state-of-the-art of deep learning-based facial image analysis in the medical context. For the consideration of patients' welfare and the development of the practice, a timely understanding of the challenges and opportunities faced by research on deep-learning-based facial image analysis is needed. To address this gap, we aim to conduct a systematic review to identify the characteristics and effects of deep learning-based facial image analysis in medical research. Insights gained from this systematic review will provide a much-needed understanding of the characteristics, challenges, as well as opportunities in deep learning-based facial image analysis applied in the contexts of disease detection, diagnosis and prognosis.
METHODS: Databases including PubMed, PsycINFO, CINAHL, IEEEXplore and Scopus will be searched for relevant studies published in English in September, 2021. Titles, abstracts and full-text articles will be screened to identify eligible articles. A manual search of the reference lists of the included articles will also be conducted. The Preferred Reporting Items for Systematic Reviews and Meta-Analyses framework was adopted to guide the systematic review process. Two reviewers will independently examine the citations and select studies for inclusion. Discrepancies will be resolved by group discussions till a consensus is reached. Data will be extracted based on the research objective and selection criteria adopted in this study. ETHICS AND DISSEMINATION: As the study is a protocol for a systematic review, ethical approval is not required. The study findings will be disseminated via peer-reviewed publications and conference presentations. PROSPERO REGISTRATION NUMBER: CRD42020196473. © Author(s) (or their employer(s)) 2021. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ.

This systematic review protocol follows the Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols guidelines.

By examining the characteristics and effects of deep learning-based facial image analysis in medical research, this systematic review bridges the gap in the literature.

This review is limited to evidence on the use and application of deep learning technologies in patients’ facial image identification and classification.

Non-English databases will not be searched, which might limit the representativeness of the results.

Background

As disease manifestations often show in various places in the human body, such as Down syndromes can change patients’ facial features, researchers have been investigating whether analysing appearance features can facilitate early disease detection and identification.1–5 One promising field is deep learning-based facial analysis.6–8 Deep learning represents a powerful range of artificial intelligence (AI) algorithm that allows computers to tackle complex problems via capitalising on neural networks, such as convolutional neural networks (CNNs), that are rich in neurons, layers and interconnectivity (see figure 1).9 Simply put, deep learning is a mechanism that allows computers to solve complex problems by neural network architecture. This ability to develop complex network structures gives deep learning a distinctive advantage: it can automatically transform raw data input into meaningful features that enable pattern identification.10 Deep learning technique has revolutionary potential in practical and research fields.11 In practice, as deep learning effectively identifies objects, traffic signs and faces, its adaptations have been widely applied in designing robots and self-driving cars.12–15 Deep learning has also been widely adopted in biomedical and clinical research, particularly in the field of medical imaging.16–19

Figure 1

Relationship between artificial intelligence, machine learning, deep learning and convolutional neural networks.

Medical conditions are often diagnosed by means of tests, such as biopsy and diagnostic imaging. An example list of diseases that have been analysed by deep learning technologies could be found in table 1. As diagnostic imaging is non-invasive and can facilitate personalised medicine, it is a preferred test option for patients and healthcare practitioners.20 21 This, in turn, has contributed to the exponential growth of medical imaging data and the increasing need for boosting medical image processing power to formulate diagnosis swiftly.21 22 Compared with traditional computer aided diagnosis for analysing medical imaging, such as hand-crafted radiomics for tumour detection, deep learning methods are superior in their ability to process large quantities of medical images accurately and cost-effectively, without exerting a heavy workload on radiologists.23–27 Evidence shows that deep learning-based medical image analysis was able to increase accuracy rates in various disease contexts, such as the identification of spinal disorder1 and lung cancer histology,28 classification of skin lesion29 and chronic gastritis,30 and the prediction of tumour-related genes31 and vascular diseases.32

Table 1

An example list of diseases that have been analysed by deep learning techniques

Disease context	Deep learning technique
Acromegaly	Convolutional neural network (along with Generalized Linear Models; K-nearest neighbors; Support Vector Machines; forests of randomized trees)63
Cancer	Convolutional neutral network64
Cornelia de Lange syndrome	DeepGestalt technology7
Coronary artery disease	Convolutional neural network65
Down syndrome	Independent component analysis66
Facial dermatological disorders	Convolutional neural network67
Keratinocytic skin cancer	Convolutional neutral network68
Inherited retinal degenerations	Convolutional neural network69
Noonan syndrome	DeepGestalt technology33
Pain intensity	Convolutional neutral network70
Neurological disorders	Convolutional neutral network71

Applying the deep learning technique to perform facial recognition and analysis tasks, researchers found that the technique yielded superior results in identifying and classifying faces of people with cancer from those without.6 Similarly, examining facial phenotypes of people with genetic disorders, findings indicate that the technique was effective and was able to yield an optimal 91% top-10 accuracy.33 Evidence further indicates that, for some tasks involving identifying and classifying facial images, deep learning techniques have often performed on par or better than human beings.5 7 10 34–36 Comparing clinical and deep learning evaluations of microdeletion syndrome facial phenotypes, researchers found that deep learning outperformed clinical evaluations in terms of sensitivity and specificity by 96%.35 These findings combined suggest that deep learning-based facial analysis technology has great potential to address complex medical challenges prevalent in healthcare. However, there has not been any systematic review on the state-of-the-art applications of deep learning-based facial analysis in non-invasively evaluating medical conditions. Therefore, to bridge this gap, we aim to systematically review the literature and identify the characteristics and effects of deep learning-based facial analysis techniques applied in medical research.

Methods and analysis

This systematic review was registered with the International Prospective Register of Systematic Reviews database or PROSPERO a priori to improve research rigour.37 38 The principles of the Preferred Reporting Items for Systematic Reviews and Meta-Analyses protocol was adopted to guide this systematic review.39 Our search strategy incorporated medical subject heading (MeSH) and keyword terms for the concept of deep learning and facial analysis. The search strategy was developed in consultation with an academic librarian, and subsequently will be deployed to target databases, including PubMed, PsycINFO, CINAHL, IEEEXplore and Scopus (table 2). The search will be initiated in September 2021. Studies will be limited to journal articles published in English. We will adopt two additional search mechanisms to locate eligible articles: (1) a manual search of the reference list of the included articles will be performed and (2) a reverse search of papers that cited articles included in the final review via Google Scholar. An academic librarian will facilitate the search process, helping administer the search and download the citation records to Rayyan (http://rayyan.qcri.org).

Table 2

Example PubMed search strategy

Concept	Search string
Deep learning	“deep learning”[MeSH] OR “deep learning”[TIAB] OR “artificial intelligence” [MeSH] OR “artificial intelligence” [TIAB] OR “machine learning”[MeSH] OR “machine learning”[TIAB] OR “convolutional neural network”[MeSH] OR “convolutional neural network”[TIAB] OR “convolutional neural networks”[TIAB]
Facial image analysis	“face detect*” OR “facial detect*” OR “face recogn*” OR “facial recogn*” OR “face extract*” or “facial extract*” OR “face analys*” OR “facial analys*” OR “face dysmorphology” OR “facial dysmorphology” OR “face phenotype*” OR “facial phenotype*” OR “face feature*” OR “facial feature*” OR “face2gene” OR “gestalt theory” OR “face photograph*” OR “facial photograph*” OR “facial expression”

Inclusion and exclusion criteria

The inclusion criteria were developed a priori and listed in table 3. Studies will be excluded articles if they (1) did not report findings on human beings (eg, studies on mice), (2) did not focus on full facial features (eg, research on retina or lip-cleft), (3) did not conduct research in a medical context (eg, in the context of criminology) and (4) did not report empirical findings (eg, editorial or comment papers).

Table 3

Study inclusion criteria

Data type	Inclusion criteria
Participants	Individuals younger or older than 18 years old
Research context	Medical research or healthcare
Analytical technique	Deep learning algorithms-based facial image analysis
Language	English
Study type	Quantitative empirical study
Outcome	Report empirical and original findings on the application of deep learning-based facial image analysis in medical context (eg, accuracy of facial image analysis in detecting Down syndrome)

Risk of bias assessment

To ensure the quality of included studies, a risk of bias assessment will be conducted independently by two reviewers, using the Cochrane Collaboration evaluation framework.40 The framework has seven domains: (1) random sequence generation, (2) allocation concealment, (3) blinding of participants and personnel, (4) blinding of outcome assessment, (5) incomplete outcome data, (6) selective reporting and (7) any other source of bias. The risk of bias will be evaluated independently by two reviewers. Potential discrepancies regarding the risk of bias will be resolved via group discussions till a consensus is reached.

Data extraction

Two reviewers will independently examine the citations and select studies for inclusion. Discrepancies will be resolved by group discussions till a consensus is reached. Data will be extracted based on the research objective and selection criteria adopted in this study. For articles that meet the inclusion criteria, the reviewers will extract the following information from the included papers: research objective/questions, disease context, sample characteristics (eg, characteristics of facial records), AI characteristics (eg, algorithm adopted), and empirical findings.

Data synthesis and analysis

If eligible studies share enough similarities to be pooled, a meta-analysis will be conducted to gain further insights into the data. Main clinical, methodological, as well as statistical differences will be carefully considered to determine the heterogeneity of the eligible studies. If eligible studies are found heterogeneous, a narrative synthesis will be conducted to summarise the data. A summary of the data extracted will be organised to synthesise key results. Both tables and graphs will be used to represent the key characteristics of eligible articles. Descriptive analysis will be performed on categorical variables. In this review, we will undertake a narrative approach to synthesise data. In other words, in addition to shedding light on key information like the sensitivity, specificity, overall accuracy of the deep learning technologies in analysing facial images (as opposed to clinicians’ analyses), we will also provide detailed analysis of the disease contexts and the techniques applied to chart the state-of-the-art of deep learning technologies in facial image analyses.

Ethics and dissemination

As the study is a protocol for a systematic review, ethical approval is not required. The study findings will be disseminated via peer-reviewed publications and conference presentations.

Patient and public involvement

The nature of the study, which is a review and analysis of previously published data, dictates that there is limited to no meaningful need for patient and public involvement in the design, delivery or dissemination of the research findings.

Discussion

Though a growing body of research has applied deep learning-based facial image analysis in the medical context for disease detection, diagnosis and prognosis, to date, no systematic review has investigated the state-of-the-art application of deep learning-based facial image analysis recognition in addressing medical diagnoses and clinical states. Therefore, to bridge this gap, we aim to systematically review the literature and present the characteristics, challenges, as well as opportunities in deep learning-based facial analysis techniques applied in medical research. To better organise the research findings, we developed a framework that illustrates the main causes for abnormal facial expressions in patients. It is important to note that we are identifying medical states and conditions and not individuals.

After reviewing the literature,5 41–55 we identified the following four preliminary categories of causes for short-term or long-term abnormal facial expressions in people: (1) gene-related factors, (2) neurological factors, (3) psychiatric conditions and (4) medication-induced triggers. Genetic-related factors, such as the presence or mutation of a certain gene, are the most studied cause for abnormal facial changes in individuals.5 7 35 Down syndrome, which is affected by the presence of a third copy of chromosome 21, is an example of genetic-related factors that can cause individuals’ abnormal facial changes.5 Neurological factors can also cause individuals’ facial phenotypes. Stroke or transient ischaemic attack is an example of neurological factors, which can occur either prior to or after the onset of the disease.56 57 The third cause for abnormal facial changes centres on individuals’ psychiatric conditions or mental illnesses, especially psychotic disorders such as the Tourette syndrome (facial tics). Last but not the least, medication-induced triggers, such as the neuroleptic malignant syndrome (caused by antipsychotic medications), can also cause abnormal facial changes in people. Details of this framework can be found in table 4. This framework will be used in the planned systematic review study to guide the data extraction process.

Table 4

Main causes for abnormal facial expressions

Cause	Definition and example
Gene-related factors	Gene-related factors are causes for individuals’ abnormal facial changes that root in the presence or mutation of one or a set of genes. Examples: Down syndrome (genetic root: presence of a third copy of chromosome 21) or Cornelia de Lange syndrome (genetic root: NIPBL or SMC1A, SMC3, RAD21 or HDAC8, BRD4 and ANKRD11 genes).5 41 42 55
Neurological factors	Neurological factors are defined as reasons that are associated with individuals’ congenital or acquired disorders of nerves and the nervous system. Neurological factors can either be related to genetic or non-genetic factors, caused by irregularity in nerves associated with the brain or the face. Examples: Neurological factors with genetic causes (eg, Rett syndrome, MECP2 gene; Cervical or Cranial dystonia, GNAL gene) and without (eg, embouchure dystonia, Oromandibular dystonia)46 47; due to nerves associated with the brain (eg, stroke) or the face (Bell’s palsy or facial paralysis, Hemifacial Spasm).43–45
Psychiatric conditions	Psychiatric conditions, especially psychotic disorders, have the potential to cause abnormal facial expressions among individuals. Psychiatric conditions could be broadly defined as mental illnesses, whereas psychotic disorder factors are causes to abnormal facial expressions that root in individuals’ impaired sense of reality. Examples: Non-drug-related Tourette syndrome (facial tics) or autism (facial expression limitation).48–50
Medication-induced triggers	Medication-induced triggers could be understood as causes to individuals’ short-term or long-term abnormal facial changes due to their adverse reactions to a certain medication of a type of medications. Examples: Neuroleptic malignant syndrome (antipsychotic drugs), tardive dyskinesia (antipsychotic medications) or drug-related Tourette syndrome.51–54

Overall, insights gained from this study will be able to provide a much-needed understanding of the characteristics, challenges, as well as opportunities in the context of deep learning-based facial image analysis technologies applied in disease detection, diagnosis and prognosis. In addition to gaining a connected and comprehensive understanding of the current application of facial image analysis, results of the study will also be able to shed light on whether, similar to facial recognition used in non-medical58 59 and medical contexts,60 61 whether or to what degree is systematic bias is present in the application of deep learning technologies for facial image analysis. A biased and inaccurate facial image analysis system will not only exert unwarranted, though avoidable, disparities on patients (eg, gender inequality),60 it will also alienate the patients from the much-needed deep-learning-assisted medical opportunities their health and well-being can benefit from.62 Therefore, for the consideration of patients’ welfare and the development of the clinical practice, a timely understanding of the scope of the research literature as well as the challenges and opportunities faced by research on deep-learning-based facial image analysis is much needed.