Pediatric chest radiograph interpretation: how far has artificial intelligence come? A systematic literature review.

Most artificial intelligence (AI) studies have focused primarily on adult imaging, with less attention to the unique aspects of pediatric imaging. The objectives of this study were to (1) identify all publicly available pediatric datasets and determine their potential utility and limitations for pediatric AI studies and (2) systematically review the literature to assess the current state of AI in pediatric chest radiograph interpretation. We searched PubMed, Web of Science and Embase to retrieve all studies from 1990 to 2021 that assessed AI for pediatric chest radiograph interpretation and abstracted the datasets used to train and test AI algorithms, approaches and performance metrics. Of 29 publicly available chest radiograph datasets, 2 datasets included solely pediatric chest radiographs, and 7 datasets included pediatric and adult patients. We identified 55 articles that implemented an AI model to interpret pediatric chest radiographs or pediatric and adult chest radiographs. Classification of chest radiographs as pneumonia was the most common application of AI, evaluated in 65% of the studies. Although many studies report high diagnostic accuracy, most algorithms were not validated on external datasets. Most AI studies for pediatric chest radiograph interpretation have focused on a limited number of diseases, and progress is hindered by a lack of large-scale pediatric chest radiograph datasets.

Introduction

Chest radiography is one of the most frequently used imaging modalities in children. It is often used to assess disorders of the pediatric lung parenchyma, airways, heart and mediastinum. Chest radiographs play a significant role in screening and diagnosing disease across all pediatric care settings, including the emergency department and inpatient settings such as neonatal and pediatric intensive care units. Chest radiographs also play a critical role in assessing support devices, including chest tube and catheter positions, and are vital in diagnosing a number of life-threatening conditions and their complications, such as pneumonia, pneumothorax and diaphragmatic hernia [1].

It is well known that the interpretation of pediatric chest radiographs differs from that in adult chest radiographs. There is a varying radiographic appearance of the normal growing child, ranging from tiny premature neonates to adolescents. Pediatric radiography often utilizes differing techniques of acquisition, and frequently, differing pathologies are encountered that are not commonly seen in adults. For example, the developmental appearance of the normal thymus in young infants can mimic a parenchymal lung infection or mediastinal mass to the unwary. Additionally, children often present with differing appearances of common pathologies encountered throughout life, such as the “round” pneumonia seen in younger children. Radiographic findings in children also have broader differential diagnoses, such as the inclusion of congenital lung disorders and other malformations [2]. These differences in radiographic interpretation pose additional challenges for any artificial intelligence (AI) models developed for chest radiography.

Background

The application of AI and its subset of machine learning (ML), especially deep learning (DL), is of growing clinical, research and commercial interest within medical imaging. Initial applications of AI have spanned all imaging modalities and now include the detection of stroke and acute intracranial hemorrhage with CT, the assessment of breast lesions with mammography and US, the detection of fractures with wrist radiographs and the assessment of a number of lung and cardiac abnormalities with chest radiographs [3]. There are many promising benefits of incorporating AI into the interpretation of chest radiographs, such as the potential for improved detection and diagnosis, quantification, triage and streamlined workflow [4, 5]. Assessment of chest imaging represents one of the most rapidly growing research areas, with determination of the presence of lung nodules, tuberculosis, pneumonia, pneumothorax and cardiomegaly being among the most common applications [4, 6]. Several excellent review articles have detailed AI and DL techniques and their applications for chest imaging; however, these reviews have primarily focused on adult chest radiograph applications [4, 7] without a dedicated summary of related benefits and limitations for pediatric patients.

Machine learning models or algorithms are trained with datasets. The designed model uses the training dataset to learn the data characteristics and then predicts the target task on new input data [8]. Dataset characteristics including size and domain, availability of labels and annotated data, class type, data balance (frequency of each class in the dataset) and the quantity and quality of features have a critical impact on the model’s performance and can be a source of bias [9]. AI training is typically done with supervision provided by labeled images. Categorical labels might be clinically based, such as the presence or absence of pneumonia, or related to radiographic findings such as consolidation or pneumothorax. Categorical labels can be derived either by separate review from radiologists or other experts or derived by natural language processing (NLP) of radiologist reports. Annotated images where individual lesions are marked with boundary boxes placed on the image provide better information for object localization and class recognition, allowing the location and size of any lesions, such as lung consolidation, to be captured. Annotated images provide a better opportunity to develop explainable AI to identify where an algorithm localizes pathology such as pneumonia using methods such as Gradient-weighted Class Activation Mapping (Grad-CAM) [10] and SHapley Additive exPlanations (SHAP) [11]. However, annotation generally requires significant expert input and time.

This review article summarizes the available literature on the application of AI to pediatric chest radiograph interpretation. Given that datasets are critical to algorithm training, testing and validation, this paper also compiles data on all publicly available chest radiograph datasets to determine pediatric image availability and potential utility and limitations for pediatric AI-related studies. We hope this paper helps researchers understand the state of AI for pediatric chest radiograph interpretation and catalyzes fruitful progress in this area.

Methods

Search strategy

We performed a systematic literature review to capture relevant studies that either designed or evaluated AI for chest radiograph interpretation in the pediatric population. We utilized the Preferred Reporting Items for Systematic Reviews and Meta-Analyses of Diagnostic Test Accuracy (PRISMA-DTA) checklist [12]. Figure 1 shows the study selection diagram. We conducted the literature search in the PubMed, Embase and Web of Science databases from Jan. 1, 1990, to July 5, 2021. Each database was queried with the following keywords: (“artificial intelligence” OR “machine learning” OR “neural networks” OR “deep learning”) AND (“pediatric” OR “child” OR “toddler” OR “infant” OR “neonate”) AND (“radiograph” OR “chest film” OR “X-rays”). Only articles written in English were considered. Two reviewers independently searched the databases to capture eligible studies (S.P. with 2 years of research fellowship experience and M.R.M., AI-electrical engineer with 6 years of research experience). They also reviewed references from selected studies to identify further relevant studies. In addition, we identified all publicly available chest radiograph datasets by analyzing the publications found in the literature search, as well as a narrative review of the databases. We also reviewed each dataset and contacted the dataset’s provider to clarify some of the dataset’s properties and availability.

Fig. 1

Study selection diagram

Study selection

Inclusion criteria were as follows: (a) studies related to developing or evaluating AI for chest radiograph interpretation and (b) articles that included a pediatric population either alone or in combination with adult patients. We screened the retrieved articles based on the title and the abstract and reviewed full texts of selected articles. Pediatric patients were defined as patients younger than 18 years based on the World Health Organization (WHO) definition of children [13]. Studies differed to some extent in their definition or inclusion of pediatric age ranges; therefore, for clarity, the tables of studies included in this review include descriptions of the age ranges. We excluded articles if they did not include pediatric patients, contained both adult and pediatric populations without separately reporting evaluation metrics for the pediatric group, did not define the age of the target population, or were not published in peer-reviewed journals (i.e. conference papers, posters, etc.).

Data abstraction

The data collected from each article included year of publication, the country in which the study originated, study design, reference standard, sample size and age distribution. In cases for which this information was not publicly available, the dataset’s provider was contacted to determine dataset characteristics and availability.

The names of all chest radiograph datasets utilized in the selected articles were also collated. Given that the datasets usually have an associated article to introduce the dataset, we reviewed the articles, access sources and metadata files of each dataset to understand the quantity and quality of the dataset, including patient inclusion criteria, available annotations and labeling, chest radiograph acquisition technique (anteroposterior or lateral views) and age range of the included patients. We collected labels used for the dataset, along with the definition of the ground truth and whether abnormal regions on each image were annotated.

Results

The initial search retrieved 2,117 articles from the databases. Because of duplication, we removed 148 articles. After assessing the title and abstract, we chose 106 papers for full-text review. We subsequently excluded 51 papers because they did not meet inclusion criteria, resulting in 55 papers being included in the study (Fig. 1).

Review of publicly available chest radiograph datasets

The 55 articles that were eligible for inclusion in the study used a total of 20 proprietary and 9 publicly available chest radiograph datasets. We found an additional seven publicly available chest radiograph databases and multiple coronavirus disease 2019 (COVID-19) datasets from other sources. Of the 16 publicly available datasets, 2 were solely pediatric [14-16] and 7 had mixed pediatric and adult populations; the others included only adult populations [17-24]. We identified a total of 20,337 unique pediatric images (5,948 normal and 14,389 abnormal) from 7 datasets, including Guangzhou Women and Children’s Medical Center (GWCMC) [14], Pneumonia Etiology Research for Child Health (PERCH), PadChest, the National Institutes of Health (NIH) NIH-14, the National Institute of Allergy and Infectious Diseases (NIAID) TB dataset, the Shenzhen database and the Montgomery County datasets. In addition, the Radiological Society of North America (RSNA) and Society for Imaging Informatics in Medicine (SIIM)–American College of Radiology (ACR) datasets include 1,276 and 617 pediatric images, respectively, which are reannotated images from the NIH-14 dataset. Figure 2 and Online Supplementary Material 1 show the age distribution in the available pediatric chest radiographs. The most frequent labels were pneumonia, infiltration, consolidation and cardiomegaly, representing 58% of the abnormal labels. Further details of the publicly available chest radiograph datasets are presented in Online Supplementary Material 2 [14-23].

Fig. 2

The age distributions of the pediatric population for the seven pediatric and mixed adult datasets containing unique pediatric images. *Note that the Radiological Society of North America (RSNA) and Society for Imaging Informatics in Medicine (SIIM)–American College of Radiology (ACR) datasets are subsets of the National Institutes of Health NIH-14 dataset

Guangzhou Women and Children’s Medical Center (GWCMC) dataset

This dataset, also known as the Kermany dataset, is from Guangzhou Women and Children’s Medical Center in Guangzhou, Guangdong, China [14]. This dataset consists of 5,856 anteroposterior (AP) chest radiographs from children ages 1–5 years. The dataset includes three labels: normal, bacterial pneumonia or viral pneumonia, including 5,232 and 624 training and test samples, respectively. Two physicians labeled all images, with a third physician verifying all test dataset labels. It is not clear what, if any, additional clinical criteria were used to determine these labels (Fig. 3).

Fig. 3

Example images from two datasets containing pediatric patients illustrate the variation in labeling used. a Anteroposterior chest radiograph of a 1-year-old boy from the National Institutes of Health (NIH) NIH-14 dataset demonstrates extensive perihilar infiltrates without effusion. This radiograph has two labels in the NIH-14 dataset: infiltration and effusion (identification number 6649). b Anteroposterior chest radiograph of a child age 1–5 years from the Guangzhou Women and Children’s Medical Center (GWCMC) dataset. This radiograph is labeled as bacterial pneumonia in the dataset. Note the presence of support tubes and lines that are not labeled (identification number person1946_bacteria_4875-Kermany). c Anteroposterior chest radiographs of a boy age 13 years from the Radiological Society of North America (RSNA) dataset. This radiograph is labeled as lung opacity and the bounding box localizes the pathology (identification number: 089a996e-425c-4,311-b473-6948b3eb1060)

Pneumonia Etiology Research for Child Health (PERCH) dataset

The PERCH project attempted to identify the etiology of severe childhood pneumonia requiring hospital admission. Children who were hospitalized with pneumonia were included from nine sites in seven countries (Bangladesh, The Gambia, Kenya, Mali, South Africa, Thailand and Zambia) between Aug. 15, 2011, and Jan. 30, 2014 [15, 16, 25]. The data are held by Johns Hopkins University and are available for research purposes under a data-sharing agreement [26]. The PERCH chest radiograph dataset consists of 3,587 images that were randomly interpreted and labeled by two members of a reading panel. This panel included 14 radiologists and pediatricians from 7 countries trained with WHO methodology. In cases of disagreement, a second group of arbitrators consisting of two experienced radiologists who were unaware of the initial assessment made the final consensus by discussion. All image readers were blinded to the clinical and demographic information associated with each chest radiograph, and readers did not interpret chest radiographs from their site. The chest radiographs were assigned with five labels, including (1) consolidation only (alveolar consolidation, including pleural effusion if present), (2) other infiltrate only, (3) both consolidation and other infiltrate, (4) normal (no consolidation or infiltrate) and (5) uninterpretable for consolidation and/or other infiltrate [15, 16].

PadChest dataset

This dataset contains 160,861 chest radiographs of 67,000 patients from the Hospital Universitario de San Juan, Alicante, Spain. Of the 160,861 chest radiographs, 5,533 are of patients younger than 18 years [17]. Cases are categorized into a hierarchical structure and annotated with 174 different radiographic findings, 19 differential diagnoses and 104 anatomical locations. NLP was used to label 73% of the cases while trained physicians manually labeled 27% from radiology reports. This dataset contains the largest number of manual labels among the large public datasets. PadChest provides some metadata, including patients’ year of birth, gender, image projection, positioning, type and manufacturer of the equipment used to acquire the chest radiograph, and annotation method (manually or by NLP).

National Institutes of Health Clinical Center (NIH)-14 dataset

The National Institutes of Health (NIH) Clinical Center released a chest radiograph dataset with 112,120 frontal-view chest radiograph images of 32,717 patients from the NIH Clinical Center [18]. There are 5,242 pediatric images in this collection, spanning ages 1–17 years (Fig. 3). Fourteen keywords of common radiologic findings were searched within the picture archiving and communication system (PACS) to extract all related radiologic reports and their associated images. NLP was employed to label the images based on the text of the radiology report. The dataset has 993 images annotated with bounding boxes to show 1,600 regions of interest for 8 labels.

Radiological Society of North America (RSNA) pneumonia dataset

The RSNA’s AI challenge in 2018 focused on pneumonia detection on chest radiographs. To facilitate this challenge, 30,227 chest radiographs from the NIH-14 dataset were relabeled by 18 radiologist volunteers from the Society of Thoracic Radiology (STR) and the RSNA. The test set was labeled based on the majority assessments of three radiologists (two from STR and one from RSNA). The pediatric subset of the RSNA dataset includes normal cases (n=462), cases with a lung opacity (n=1,074) and cases with no lung opacity but another abnormality (n=429), for a total of 1,965 pediatric images. Cases with lung opacities were manually labeled with bounding boxes to highlight regions of opacity [19, 27] (Fig. 3). This dataset has an age range of 1–17 years. Training and test sets are provided by the challenge holder.

Society for Imaging Informatics in Medicine (SIIM)–American College of Radiology (ACR) pneumothorax segmentation dataset

The SIIM and the ACR created a dataset for a pneumothorax detection challenge. The dataset is a subset of the NIH-14 dataset and consists of a 12,047-image training set. Because the dataset has been published for competition purposes, the test set is not publicly available. Six radiologists from SIIM and 13 radiologists from ACR were involved in labeling each image manually [21, 28]. The dataset contains 617 pediatric cases ranging 1–17 years old. This dataset also provides training and test sets.

National Institute of Allergy and Infectious Diseases (NIAID) tuberculosis dataset

The United States National Institute of Allergy and Infectious Diseases (NIAID) led the development of the Tuberculosis Portals Program, an international collaboration for tuberculosis data-sharing and analysis to promote tuberculosis research. It has a database containing clinical, imaging and bacterial genomic information from drug-sensitive and resistant tuberculosis cases sourced from 40 sites across different countries. The dataset contains 6,251 chest radiographs and CT images from members of the collaboration. Among these, 4,656 images are unique images from 3,149 unique patients and have been labeled manually or by a convolutional neural network (CNN) classifier. There are 71 chest radiographs from 45 pediatric patients in total [23].

Shenzhen dataset

This dataset was obtained from the outpatient clinics of Shenzhen No. 3 People’s Hospital, Guangdong Medical College, Shenzhen, China, over 1 month in 2012. Associated available metadata include gender, age, radiologic diagnosis and, in some cases, the location of the abnormality. However, the annotation method and ground truth of labeling were not reported [24]. The dataset includes 662 chest radiographs, including 31 pediatric chest radiographs (from children ages 2 months to 17 years).

Montgomery County dataset

This dataset was collected from the tuberculosis screening program in Montgomery County, MD, United States. It comprises 58 chest radiographs of tuberculosis and 80 normal chest radiographs, with only 17 images of pediatric patients (ranging 4–17 years old). Images in the dataset are associated with a segmentation mask to illustrate the involved lung region as well as metadata including gender, age, abnormality seen in the lung, diagnosis and location of abnormalities. Some cases also contain labels indicating chronic or acute tuberculosis patterns and the treatment received and duration [24].

Review of artificial intelligence studies for pediatric chest radiographs

The comprehensive literature review showed that 36 (65%) of the 55 eligible studies published between 1990 and 2021 focused predominantly on pneumonia. Of the studies focusing on pneumonia, 32 used publicly available datasets and 4 used private datasets. Beyond pneumonia, other studies have focused on detecting and classifying thoracic diseases such as hyaline membrane disease and meconium aspiration syndrome, quantitative and scoring prediction, catheter and tube detection and classification, and lung segmentation (Fig. 4).

Fig. 4

Chart shows a summary of conditions studied in the 55 selected pediatric artificial intelligence and chest radiograph articles, listing the total numbers and percentages

Studies to detect pneumonia on pediatric chest radiographs

Online Supplementary Materials 3 and 4 summarize reviewed papers pertaining to detecting pneumonia on pediatric chest radiographs [14, 26, 29–62]. Most early studies on the classification of pediatric chest radiographs as pneumonia being present vs. absent used conventional texture analysis and conventional machine learning classifiers such as k-nearest neighbors [29] and support vector machines [30].

More recently, deep learning has become the dominant ML approach for chest radiograph classification for pneumonia, with a variety of CNNs used over the last few years. While larger datasets have generally been required for training deep learning algorithms compared to conventional texture analysis, many deep learning algorithms have achieved higher performance. Kermany et al. [14] employed an Inception v3 architecture to classify optical coherence tomography (OCT) images of the retina and subsequently demonstrated the generalizability of this technique for the classification of pediatric chest radiographs. They achieved area under the receiver operating characteristic curve (AUC) of 0.968 for classifying radiographs as pneumonia present vs. absent. The GWCMC dataset was first released in this project.

Since the GWCMC dataset was released, multiple additional studies have used this dataset for classifying pediatric chest radiographs as pneumonia present vs. absent, or bacterial pneumonia vs. viral pneumonia vs. normal, using different architectures such as AlexNet, ResNet, DenseNet, SqueezeNet, VGG16, VGG19, ResNet-18, ResNet-50 and Inception v3 (Online Supplementary Material 4) [14, 26, 33–62].

In addition to pre-training models with general image datasets such as ImageNet (which is commonly employed in many algorithms in medical imaging [63]), pre-training models with large datasets of adult chest radiographs provide a promising approach to reducing the number of pediatric chest radiographs required for training [37, 54]. Tang et al. [54] used GoogleNet (Inception v3) to classify pediatric chest radiographs from the GWCMC dataset as pneumonia present vs. absent. When trained on an adult dataset (NIH ChestX-ray 14), the model achieved an AUC of 0.916 in the test subset of the GWCMC dataset. When the same model was trained exclusively on a subset of the GWCMC dataset, the model achieved an AUC of 0.975, and when the same architecture was pre-trained on NIH ChestX-ray 14 and then fine-tuned on a training subset of the GWCMC dataset using a transfer learning approach, AUC increased to 0.985.

Ensemble approaches have also shown excellent results. Chouhan et al. [45] individually trained AlexNet, DenseNet121, Inception v3, GoogleNet 50 and ResNet-18 on a training subset from the GWCMC dataset and subsequently developed an ensemble model by majority voting; accuracy of 96.4% was achieved, higher than each of the individual models. A similar approach was utilized by Hashmi et al. [46], who achieved an accuracy of 98.4% and an AUC of 0.999. Other groups have similarly combined well-known CNNs and conventional machine learning classifiers, as well as combined handcrafted features from texture analysis and features extracted from CNNs, with promising results [60, 61].

Qu et al. [55] highlighted the effect of class imbalance on pneumonia present vs. absent classification performance. They utilized ResNet-18 and showed that a close-to-balanced training set (with relatively equal numbers of images with pneumonia present or absent) resulted in the best performance. Furthermore, oversampling and under-sampling methods were successful in mitigating class imbalance.

To capture multi-scale image features while minimizing computational costs, Hu et al. [38] developed a multi-kernel depth-wise convolution (MD-Conv). In addition to assessing the network using NIH-14, they assessed the network for classifying pediatric chest radiographs using the GWCMC dataset and achieved an AUC of 0.98, slightly higher than the original paper of Kermany et al. [14], which achieved an AUC of 0.96.

Studies using artificial intelligence to detect other pathologies

Few AI studies have considered the broad range of pathologies other than pneumonia that are common on pediatric chest radiographs (Online Supplementary Materials 5 and 6) [64-82]. A pioneering study by Gross et al. [64] first implemented a neural network to assist in neonatal chest radiograph interpretation. Zaglam et al. [65] and Chen et al. [66] used machine learning models to classify chest radiographs of children for respiratory distress syndrome, bronchiolitis/bronchitis, bronchopneumonia/interstitial pneumonitis, lobar pneumonia or pneumothorax.

Artificial intelligence has also been used to automate the quantitative scoring of pediatric chest radiographs. Toba et al. [71] achieved an accuracy of 86% and AUC 0.88 in predicting the pulmonary-to-systemic blood flow ratio based on chest radiographs. These performance metrics were equal to or higher than three pediatric cardiologists’ predictions. Predicting Brasfield chest radiographic score in children with cystic fibrosis is another AI use case that has been shown to achieve similar performance to radiologist scores [72]. A combination of an artificial neural network and texture analysis was performed by Clark et al. [76] to classify pediatric chest radiographs of children with cystic fibrosis. A strength of the study is its comparison to CT images as a gold standard, but a substantial limitation — both for training and testing the algorithm — is the small dataset.

Artificial intelligence for catheter and tube assessment — particularly for neonatal radiographs — is another important yet underdeveloped area for AI in pediatric imaging. Various techniques, including image segmentation using deep learning [73] and image classification based on a CNN [80], are used to detect and classify tubes and catheters. AI has also been used extensively for lung segmentation, both as a pre-processing step to increase the accuracy of other classification models focused on lung pathology, as well as a method of estimating lung capacity. Mansoor et al. [74] developed a deep-learning-based method of lung segmentation generalizable to both adult and pediatric chest radiographs for the purpose of identifying lung fields. Their algorithm achieved Dice similarity coefficients, which measure the overlap between the ground truth and model segmented regions, ranging from 0.969 to 0.975. Other pediatric chest radiograph datasets, including the GWCMC dataset and a pediatric chest radiograph dataset from the University of Michigan, have also been used in training and testing algorithms for lung segmentation [66, 83]. It has been shown that applying age-specific models (using age groupings of 0–23 months, 2–10 years and 11–18 years), instead of adopting a model for all ages, improves algorithm performance [84].

Beyond image interpretation, efforts have focused on applications of AI that might be useful in PACS. Hržić et al. [77] used a transfer-learning approach based on ResNet, VGG16, and AlexNet to correct the orientation and alignment of radiographic images, achieving 99.3% accuracy. Kim et al. [69] performed classification of anteroposterior and posteroanterior chest radiographs by training a CNN. The CNN that was trained on both adult and pediatric chest radiographs achieved AUCs ranging from 0.985 to 0.999 when tested on three pediatric datasets — two of which were external. The CNN trained using only pediatric chest radiographs achieved an AUC of only 0.997, possibly because of the substantially smaller training dataset on which the CNN was trained.

Discussion

Chest radiography is one of the most frequently performed imaging tests, being easy to perform, low in cost and readily available even in low-resource settings. It provides important information regarding pediatric thoracic pathology; however, chest radiographs can be challenging to interpret, with significant variability and discrepancy in findings often noted. Machine learning, especially deep learning, is a rapidly developing field with great promise for medical imaging, offering the potential for enhanced accuracy, faster diagnoses and standardized interpretations. Specific AI applications for chest radiography have rapidly proliferated within the last 5 years, with several commercial products for abnormality detection, including detection of lung nodules and cancer, pneumonia and pneumothorax, among others now available [3]. Despite this activity, there has been limited application of machine learning to pediatric chest radiography.

The availability of appropriate pediatric image datasets is critical for training, testing and validating AI algorithms in pediatric radiology. Indeed, the performance of any model is dependent on the dataset used for training. Our systematic review found a paucity of gold standard open-source datasets for pediatric ML training and testing. We found a limited number of pediatric-specific datasets with an overall quite limited number and age-limited pediatric images available. The development and public release of the GWCMC dataset [14] can be considered a turning point that has brought AI researchers’ attention to the pediatric population. The GWCMC dataset has resulted in the generation of multiple studies predominantly exploring the diagnosis of pediatric pneumonia using deep learning. However, several studies had been conducted on pediatrics before the release of this dataset; most of these studies utilized traditional machine learning techniques and generally small size and proprietary datasets. In the GWCMC dataset, the ground truth was established using only radiologic features, without using clinical and paraclinical information. On the other hand, the PERCH chest radiograph dataset has strength in this area and includes infants, with well-defined patient inclusion criteria, multi-institutional data sources and associated clinical and paraclinical information [15, 16]. It is hoped that making this dataset available for AI research and development will mark another milestone in pediatric imaging AI development. There is a need for additional open-source pediatric-specific datasets that include a broader range of children across different age ranges and across additional pediatric conditions and diseases.

Important considerations for datasets include whether they were retrospectively or prospectively collected, the reference standards used, and how labels and annotations were obtained. For example, labels in the NIH training set were obtained by natural language processing, which is considered to represent weak or noisy supervision in the training process. Labeling images based on NLP from radiology reports is intrinsically associated with uncertainty [85] because radiologists might mention a disease only as a possibility, not as a definitive diagnosis (Fig. 3). The assembled datasets sometimes have poorly defined criteria for the presence or absence of the disease and might not reflect the uncertainty of radiologists’ diagnoses for some cases. It can be costly and time-consuming to create large-scale, richly labeled datasets because this generally requires extensive effort by experts. Appropriate datasets are not yet available for many diseases, especially for the pediatric population. Only a small pool of pediatric imagers is available, making obtaining these datasets a multi-institutional challenge. The use of transfer learning might assist in addressing the small size of available pediatric datasets and help reduce computation time for training models.

Our literature review on AI for pediatric chest radiograph interpretation found pneumonia detection to be by far the most common task for which AI algorithms have been developed. This is a particularly important use case in pediatric radiology, given that pneumonia is one of the leading causes of morbidity and mortality in children, accounting for an estimated 900,000 deaths worldwide annually [86, 87]. The diagnosis of pneumonia is essentially clinical, and radiographic findings such as consolidation, infiltration or pleural effusion only support the diagnosis [88]. However, many images of the large-scale public chest radiograph datasets have been labeled with just “pneumonia” [17, 18, 85]. When labeling images with clinical diagnoses rather than radiographic observations, it is important to ensure that both radiographic observations and clinical information are incorporated when images are labeled, or that the images are obtained from a population of children with a high pre-test probability of having that diagnosis. In addition, many algorithms in our review were trained only by the image features present without incorporating any clinical information as inputs. The drawback of this approach is that the AI algorithm might provide a clinical diagnosis that does not fit with the child’s clinical presentation as an output, even if it is consistent with the imaging findings. Therefore, we recommend that researchers consider the distinction between radiographic observations and clinical diagnoses when labeling images in future chest radiograph dataset construction.

The challenges of developing pediatric AI models are not just limited to the lack of appropriate datasets. Recognition of what is considered normal is defined differently in the various datasets identified in this review. The chest wall and lungs, ossification centers and cardiothymic width grow and change during normal development. This makes some tasks such as lung segmentation more difficult for AI models, and separate models for specific age groups might be required. Children with medical devices, including a variety of chest tubes and catheters and pacemakers, are often included within the group called normal. In a scenario that a chest radiograph demonstrates catheters, tubes or other medical devices, the model might easily find the device as an abnormality, leading to a high false-positive rate. Care must be taken to mitigate the bias that this can introduce into the results.

Another challenge is the need for the integration of clinical data. For example, neonatal pneumonia and meconium aspiration syndrome can have similar radiographic observations, and only knowledge of a specific patient’s clinical data makes differentiation possible [89]. An additional concern is a fact that specific pediatric age groups have differing disease predispositions. For example, transient tachypnea of the newborn is diagnosed only in neonates [90]. Studies have demonstrated that AI models might show false positives in cases with insufficient inflation, inappropriate positioning, non-standard exposure, clothing, or external or implanted medical devices [5, 91]. These challenges can be particularly problematic in pediatric populations because providing optimum conditions for taking high-quality images in pediatrics can be difficult [72]. Different radiologists have varying levels of sensitivity and specificity, which they alter to fit the clinical situation. In the diagnosis of lung nodules in children with a known malignancy, for example, high sensitivity is essential, but sensitivity can be reduced to boost specificity in the identification of incidental pulmonary nodules in asymptomatic individuals. In children with established immunodeficiency, high sensitivity is essential, while a greater specificity is preferred in the case of community-acquired respiratory infections and fevers of unknown origin. Similarly, different operating points could be defined for AI algorithms based on clinical information, though this work has not been undertaken.

Deep learning models in radiology frequently suffer from generalization issues because of large source and target domain divergence and sometimes need fine-tuning for target domains, including patient age, as in pediatrics. For example, Tang et al. [54] showed that there is a significant domain shift between adult and pediatric datasets, as demonstrated by substantially higher AUC values for a model trained on a pediatric dataset compared to the same CNN architecture trained on an adult dataset when both models were tested on a pediatric dataset. Transfer learning can learn the common characteristics of both domains, leading to a better initialization of the model parameters and faster training.

Some reporting guidelines have recently proposed specific AI extensions to develop and report AI-related studies, including the Standards for Reporting of Diagnostic Accuracy Studies (STARD) [92], Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) [93] and Checklist for Artificial Intelligence in Medical Imaging (CLAIM) [94]. These guidelines provide enhanced transparency and reproducibility and help readers assess the generalizability of results. Many studies included in this systematic review lack at least some of the reporting items recommended in these guidelines. In this review we found a wide variety of machine learning architectures used with significant study heterogeneity in methodologies, sizes of datasets for training and testing, validation techniques, reference standards, terminology and reporting, which is not surprising given the time period covered by this review and the recent technological progress. Many studies reported high diagnostic accuracy; however, there was often poor study design, which can lead to bias and overestimation of the accuracy of these algorithms. Some of these study limitations are noted in the tables provided in Online Supplementary Material. For example, many of the studies summarized here did not report their performance indices’ average and standard deviation. Furthermore, according to the prevalence of abnormalities, some indices like AUC, F1-score and Matthews correlation coefficient (MCC) [95] should be reported in addition to accuracy but are often not reported. Clinical trials involving AI and DL interventions with rigorous standardized reporting guidelines are expected to be valuable in determining the future clinical implications of algorithms in real life [54]. According to the STARD checklist, confidence intervals give information about a range within which the real value is likely to be found, as well as the direction and strength of the observed effect. This allows for conclusions to be formed regarding the study’s statistical validity and clinical usefulness. In addition, full access to the dataset and code is unavailable for the vast majority of AI studies, which limits the reproducibility of AI research that has been published [95]. These limitations in pediatric AI research make progress in this domain more challenging.

The performance of DL models as currently reported in the literature might not reflect the actual performance of the models in clinical practice. For example, despite recommendations that an external test dataset be used, almost all of the studies identified in this review used the same dataset for training, testing and validation. Algorithm performance might be significantly different when used in clinical settings with different patient populations, imaging equipment and acquisition parameters. Further, the performance of most algorithms included in our review was not compared with the performance of radiologists.

In radiology, AI products can be used for triage or alongside a radiologist as a second reader [96]. While commercial software is increasingly becoming available for adult imaging, our review highlights that progress for pediatric chest radiographs interpretation lags. Many commercially available United States Food and Drug Administration (FDA) approved/cleared tools are not at this time applicable to pediatric populations [97, 98]. In addition to common use cases such as pneumonia detection, there are a number of distinct use cases for AI development in pediatric imaging, such as neonatal catheter and tube assessment and cystic fibrosis scoring. How best to integrate the output of AI into clinical practice and workflow remain unanswered questions.

There are many opportunities to develop AI for pediatric imaging. Use cases range from correctly orienting images in PACS, flagging abnormal images to prioritize them for interpretation by a radiologist (particularly for life-threatening conditions like pneumothorax), automatic labeling of images to support the creation of annotated datasets for development of other AI algorithms, automatically detecting radiographic features (“lung sequencing”) and localizing potential abnormalities in the various lobes, performing automatic scoring of diffuse lung abnormalities (such as for cystic fibrosis) and providing preliminary interpretations to referring clinicians in resource-limited settings where many radiographs go unreported.

This systematic review has several limitations. First, our literature search strategy did not encompass conference papers or articles that were only included in open-access repositories such as arXiv, resulting in some pediatric AI studies likely being missed. Second, while we performed a systematic review, we did not perform a meta-analysis as part of this study. This is a result of the rapid evolution of AI approaches over the last decade, the fact that most studies did not assess AI algorithms on external datasets, and the substantial heterogeneity of the studies included (including heterogeneity in terms of patient population, classification approaches used and classification tasks). Third, while we considered the strengths and limitations of each study included in the systematic review, we did not use a validated checklist to assess study quality. As AI in pediatric chest radiographs interpretation continues to advance and a growing body of clinical evidence is established, formally assessing study quality will be increasingly important as AI is considered for clinical use in pediatric radiology.

Conclusion

In this article, we presented a systematic review of the application of AI to pediatric chest radiograph interpretation. Although ML in pediatric radiology holds great promise, it clearly remains in its infancy and has a long way to go before adding value to the pediatric radiology workflow. Designing appropriate AI models requires significant work with reliable features derived from diagnostic images. Radiologists with knowledge of AI bring ideas to solve problems and act as a link between radiology and biomedical engineering. For AI to progress in pediatric radiology, they suggest improving the awareness of radiologists about AI science, providing better publicly available image datasets, and creating developmental research to design AI models and validate AI’s utility and reliability in the clinical workflow.

(DOCX 54 kb)

(DOCX 58 kb)

(DOCX 55 kb)

(DOCX 61 kb)

(DOCX 65 kb)

(DOCX 54 kb)

(DOCX 15 kb)