Effective Utilization of Multiple Convolutional Neural Networks for Chest X-Ray Classification.

Out of the numerous types of Medical Imaging modalities available, radiography stands out a bit more than others due to its capabilities of diagnosing diseases and conditions, including life-threatening conditions. Its affordability is another main reason for its prevalence. Chest Radiography holds even higher importance, as it focuses a critical area of the human body. However, interpreting a Chest Radiography image can be challenging and usually done by an experienced Radiologist for accurate results. There are two main issues related to this. One is that in some countries, experienced Radiologists are scarce. The other issue is that the inevitability of human errors in diagnoses. Researchers attempt to use Artificial Intelligence to address these two issues. Most of the existing work incorporates Convolutional Neural Networks for this purpose. This paper presents a novel way of parallelizing multiple architectures of Convolutional Neural Networks focusing on Chest X-ray classification. The paper further presents a comprehensive evaluation of the existing architectures with the parallelized results of them using our method. We used four large-scale datasets, including a non-medical one, for the evaluation of our models. We managed to achieve better accuracy for 9 out 13 and 11 out of 14 labels on our two main evaluation datasets. The paper concludes by presenting the limitations and future improvements possible for the system.

Introduction

Background

Imaging for medical purposes (Medical Imaging) involves various technologies and processes that capture different areas/organs of the human or the animal body. These imaging modalities contain various types of visual representations corresponding to different structural and qualitative properties of the scanning area. These are very helpful in identifying and confirming a disease’s presence or keeping track of the progression of an already diagnosed disease. Radiography, Magnetic resonance imaging (MRI) and Ultrasonography are typical examples of Medical Imaging Technologies.

A Report by UNICEF in 2019 [1] shows that a child dies because of pneumonia every 39 s. It further shows that pneumonia causes more deaths, especially in children more than any other infectious disease. Tuberculosis is another dangerous disease like that. Even though deadly diseases like Pneumothorax have a comparatively low death rate, [2] shows that it has a high recurrence rate which can be very harmful. Most chest-related diseases can be controlled or sometimes completely cured if identified early and treated well. The most used method in diagnosing chest-related diseases is Chest Radiography. The advantages of chest radiography are the low cost and its convenient operation. Nevertheless, as shown in [3], confusions can occur when diagnosing chest-related diseases using chest radiography due to the complex structural representations in chest radiography images, which can differ from one person to another.

Problem Definition

A report released by Association of American Medical Colleges (AAMC) in June 2020 [4] predicts that by 2032 there will be a shortage of physicians including radiologists of about 122,000. Similarly, as per a report released in 2019 [5] by the Royal College of Radiologists (RCR) shows that only 20% of all the ‘trust and health’ boards in UK has sufficient radiologists required to perform emergency procedures.

Furthermore, poorly resourced areas and countries may not have quality and well-experienced radiology services. In a study in 2015 [6], Rwanda, a country with a population of more than 12 million, only had 11 active radiologists for the whole country. Comparably, Liberia being a country with more than 4 million population, had only two active radiologists till 2015 as per [7]. Moreover, [8] shows that in the United States, the percentage of active radiologists out of the whole medical professionals have been decreasing in the previous years. Reference [9] shows the effect of the insufficiency of the radiologists in the United Kingdom. It also shows the adverse financial effects of insufficient radiologists by noting the overtime paying and the outsourcing costs that the UK government had to bore.

Considering the information provided in the previous sections, it is conclusive that there is an evident insufficiency of professionals with radiographic knowledge in the world. Also, as radiography screening is highly dependent on the experience of the professionals, a guide or a support system would be very helpful even for the experts.

Motivation

With the advancements in hardware and software technologies, researchers tried to use computing to identify diseases in CXRs. The first attempt for a CAD system was found in the 1960s [10]. Then with the publicly released datasets like [11, 12] and [13], these types of researches have been boosted recently. However, systems built using datasets like those, are currently at the research level. Although they have performed well to some extent, issues like generalization and practicality are still implied. Another main issue is that the current literature primarily focuses on building a model to get higher accuracy levels but not how it will perform in practical medical scenarios. For example, [12]’s algorithm seems promising with the Area Under the Curve (AUC) scores presented in the paper. However, it has been only tested for few pathologies wherein practical scenarios multiple pathologies may need to be identified in a CXR [14].

Contributions of this Work

This paper presents novel Neural Network Architecture, which will accurately detect a selected set of pathologies in a Chest X-ray. This research will also focus on a thorough re-evaluation and critical analysis of the existing works and to present them to the research community in a sensible manner. Below we list the main contributions of this research.

A critical evaluation of the existing work related to Computer-Aided Detection systems. (We suggest referring to our initial paper [14] for a more broader and critical discussion on the existing works)

A requirement analysis for the proposing system via a brief survey study.

Parallelize Block: A novel way to utilize multiple Convolutional Neural Network (CNN) models, especially in a case of classifying multiple classes, which the said CNN architectures are specifically better at.

ParallelXNet: A CNN-based model trained on multiple datasets freely available to the community.

A Conceptual Framework for Parallelizing CNN models.

A systematic re-evaluation of main CNN architectures, comparing them, utilizing all the three main freely available datasets.

The issue of generalization of the models, proved quantitatively, which existing literature only have hinted at.

Paper Structure

We write this paper as a follow-up paper for our previous publication [14], addressing a few of the critical gaps there. The paper starts with an introduction, providing the necessary background knowledge. Then in the second section, we concisely discuss the existing work that has been done related to our research. In the third section, we present a brief requirement analysis on our research topic. In the fourth section we present our novel CNN architecture with its mathematical explanation. Section six and seven presents the test results, limitations, and the future work.

A Concise Review of Existing Work

The subject ‘Medical Imaging’ is a vast field that involves various different methodologies in capturing the inside body of living beings. Different imaging technologies output a different set of visual representations, which corresponds to a specific set of physical qualities and behaviors of the observing areas of the living body. Hence, this helps to identify and diagnose diseases commonly linked with the physical qualities and behaviors of the living bodies and their organs. These visual-based modalities allowed Computer Vision and Artificial Intelligence (AI) to support the diagnosis process. We suggest referring to our initial paper [14] for a broader and critical discussion on the existing works.

AI for Medical Diagnosis

In the literature, the first attempt to build a Computer-Aided Diagnosis (CAD) system was in 1963 [12]. It was to support the diagnosis of Lung Cancer from Roentgen Images (a.k.a. X-ray Images). However, with the world’s technology back in the day, it was nearly impossible to build an advanced CAD system to equalize or surpass human abilities. However, with the rapid development of the technologies, those systems have seemingly become possible to implement during the recent past. We also note that being ‘possible’ does not necessarily mean ‘easy-to-build’. AI-based CAD systems should be initiated with a good research foundation. Researchers interested in Medical AI have been working hard to build those type of systems. Lung nodule detection [15], eye disease diagnosis [16], cancer detection [17], and localization are some outcomes due to that effort. Research focused on utilizing Deep Learning to tackle COVID-19 like [18, 19] are some notable research carried out recently.

The study in [20] has evidently shown that incorporating CAD systems for the diagnosis of pulmonary nodules raised the diagnosing accuracy of the radiologist by more than 4% average. We believe further quantitative studies should be done in identifying the effect of utilizing AI for Medical Diagnosis and its advantages.

AI for Chest X-Ray (CXR) Diagnosis

As per our initial paper [14] we mainly identify three components that we should consider when discussing an AI-based CXR diagnosis system.

CXR Labelers

Image Preprocessing

AI algorithm; Below we briefly look into each of these component.

CXR Labelers

The result of a CXR diagnosis by a radiologist commonly comes as a narrative report called ‘Radiology Report’. Radiologists briefly write their observations in the report intending for another specialized medical professional or a physician to get a good knowledge of what is being displayed in the CXR image. As per [21], the sections may include the following; Title, Indication, Procedure, Findings, Impression and Footnotes. It further points out a survey that shows one out of two physicians directly looks only into the ‘Impressions’ section of a radiologist’s report implying that section would have higher importance when understanding a radiology report.

Commonly, labelers use Natural Language Processing (NLP) techniques on top of the Text Mining techniques to extract information out of the radiology reports. Labelers [22, 23] were built focusing more on clinical narrative-styled contents and biomedical publications, including Patient Discharge summaries, Radiology reports and Electrocardiograms (ECGs). As noted in [22], it is more biased and performs well for medical texts like Patient Discharge Summaries. However, [22] is a Machine learning (ML) based tool where [23] is built using Ontology-based methodologies. Reference [11] have combines both ML and Ontology-based methods and have quantitively shown an increment in accuracy.

However, none of the mentioned methods were capable in identifying the sense of ‘uncertainty’ which is quite frequent to find in radiology reports as per [24]. They were able to adapt the method in [23] to detect possible ‘uncertainty’ implications in radiology reports. Later, [12] and [13] introduced even advanced and better-performing labelers to label their own CXR datasets.

Image Preprocessing

Similar to the general DL-based classification models, Image Preprocessing techniques also play a major role in this use case. This has become essential for a DL-based Image Classification system as the images we collect to train the models does not always come precisely in the form we want. As the models see image datasets as typical numerical data distributions, it is essential to make sure the data distribution is in a good form which makes easy for the model to get used to and learn from it. Below we list out the main techniques that are related to this research problem. However, this paper does not explore these techniques in-depth as we do not focus much on improving them but on utilizing them.

Cleaning

Enhancement

Segmentation

Augmentation

Reference [25] proposes a comprehensive way to do data cleaning identifying the image samples that are out of the distribution. In the literature, image segmentation related to CXR images was done mainly in three ways; Rule-based, ML Based and DL Based. References [26] and [27] are examples for some notable work done related to this. Reference [28] presents reliable results segmenting the lung area of CXR images nearing human-level performances.

AI Algorithm

AI Algorithm can be called the core of any AI-based system. Back in the day, researchers used ML-based algorithms in common. However, with the technological improvements and the availability of large-scale datasets, DL-based algorithms easily surpassed ML-based algorithms. Our paper [14] contains a more descriptive discussion on ML-based algorithms and their limitations. In this paper, we briefly discuss only the notable existing work done using DL-based approaches.

It is significant that the evidence found in literature repeatedly prove the ability of DenseNet architecture [29] and also the ResNet architecture [30] for accurately identifying specifically the pathologies present in Chest X-rays. After an in-depth investigation, we identified that the identity mappings of them are the main contributing factor towards this behavior. These identity mappings evidently work very well in fitting CXR images which have structurally very similar visual model across all the samples. This is due to the visually consistent structure of the human upper skeleton. Even though CNN architectures are capable of learning just by image data and their related labels, evidence like [31] in the literature suggests they perform better when we influence them positively to learn on the CXR images. Below we briefly discuss exiting approaches directly related to our research. Most of these research use datasets in [11] and [12] except for the work done in [32], which uses the largest openly available dataset presently, introduced in [13].

Reference [33] utilizes the squeeze and excitation blocks (SEnet) published in [34] into their Neural Network. They have used a multi-map transfer layer and max–min pooling methodology to deal with different pathologies separately. However, for few diseases like Hernia, AUC scores have been reduced due to the usage of SE blocks, max–min pooling and multi-map layers. Also, test time augmentation (TTA) techniques have been heavily used, generating six augmentations from each test image sample. Little to no evidence has been depicted on the quantitative effect of these TTA techniques on the final test results evaluation.

Reference [31] have incorporated lung segmentation methodologies to force the CNN network to focus more on the lung area in the predictions making process. However, little to no improvement can be seen in AUC scores, even with heavy computations compared to [35] that uses the DenseNet-121 network with very few modifications.

Reference [36] proposes a branched architecture with three main branches. Firstly, a CNN network tries to localize the abnormality. Then that area is passed through again in a CNN network. Then the detection results taken from both the branches were combined to get the final result. However, this makes the whole architecture highly dependent on the CNN network that tries to localize the abnormality. Most importantly, even though the paper presents a state-of-the-art result, they have used a custom split for the evaluation. This makes the results incomparable to the other results found in the literature. Also, the split does not guarantee that the patient overlapping would not be present between the splits.

The approach presented in [37] is currently at the 2nd position in the CheXpert [38] competition. They have used model ensembling and Label Smoothing Regularization (LSR) in their approach. They have also used a novel approach called Conditional Training (CT) methodology, which uses hierarchical dependencies among different pathologies to each other to improve the predictability of the model. However, the evaluation of the model was only done on five pathologies out of twelve found in the training split of the dataset. Furthermore, the authors have acknowledged the possible inabilities of the CT method to generalize well on different distributions. Hence, this would possibly make the detection results prejudice to the distribution that training data is originated from.

Work [32] have designed a CNN-based architecture specialized for CXRs to take the two views of one of them, ‘frontal and lateral’, and work with them concurrently for making the final detection result. However, the presented AUC scores for each pathology are relatively low.

Most of the CXR datasets used by the aforementioned research studies were compiled from geographically neighboring sources. Due to this fact, the models trained on those datasets may not perform up to the exhibited evaluation results in reality. This could be due to hidden and unwanted patterns due to the equipment used, medical conditions of the belonging patients, followed imaging procedures and technologies. Furthermore, almost all the existing works merely rely on the CXR image (or more views whenever possible) to detect pathologies on CXRs. However, in reality, medical professionals consider various biological factors, including but not limited to; symptoms, age, gender, geographical environment, behavioral patterns, medical history and various other clinical and non-clinical tests.

End-to-End Systems

Table 1 shows similar systems currently available. It can be seen, even though there are clinically certified systems for scanning reports like ultrasound scanning and CT scanning [39], CADs for CXR are still under very preliminary stages and have never been ‘certified clinically’ for medical usage. Although attempts like [40] show good accuracies in the publications, the datasets used are not made available publicly and it is unclear whether the algorithms are actually effective or whether that the ‘amount of data’ used to train or test is what causing the good accuracy scores.

Table 1

AUC scores when tested on MIMIC-CXR 2020 dataset

System	Description
Oxipit [41]	Claims that ‘No finding’ X-rays can be identified very accurately and claims that the system covers most of the radiological findings
	However, no research has been found published on the accuracy of the performance of the system
	Only used for research purposes. No clinical research has been done to date
	ISO certification was awarded, but only for information security
qXR [40]	Trained on over two million radiographic images, and trained mostly focused on tuberculosis [42]
	Published results show high accuracies for abnormalities related to tuberculosis and focus more on CT [42]
	Claims the ability to work with MRIs as well
	No certification has been awarded to date

Survey Study

Questionnaire

A Questionnaire was distributed among a limited, targeted audience. The complete questionnaire is available in the appendices. The targeted audience was the Medical Students in their final study year (majority) of MBBS degree in Sri Lanka from different state universities. Fifty-three participants have participated in this preliminary survey. Below is the analysis of the results of the questionnaire. As interviews ("Interviews") were focused on the field experts, the questionnaire was circulated among the practicing medical students to get a better understanding of high-level requirements and the comments on them for the proposing system. This also helped the us to understand how it will support medical education and in a practicing environment. See Appendix for the complete Questionnaire results.

The majority (77%) of the responses come from final year (fifth year) students following an MBBS degree. They are the most targeted audience in the questionnaire as generally, they are the most experienced medical students.

It could be seen that, very often, medical students get to diagnose patients by their CXR. However, with the responses to the 5th question, it is clear that for further confirmation, a more specialized medical test might be necessary in most cases to confirm their findings. This was also noted in the interviews.

Most of the respondents disclosed that they further refer to a specialized doctor or a colleague to get their findings to confirm. Hence the system might potentially help as a support system.

From questions 7 and 8, it can be stated that even though they are optimistic about the system, at the same time, they are uncertain whether the system would be adequate to overcome their errors. Hence the system must be designed in a way to gain their confidence having a provable accuracy.

From the 10th and 11th questions, a clear majority expresses the importance of accuracy in CXR screening over its urgency.

From the general comments of the questionnaire, the majority seems to have a positive thought about the system. However, one respondent (medical student) has suggested that the system might be helpful for radiologists but can be misleading for physicians. Further, the respondent suggests that it is better not to give medical students the system as they should learn to screen CXRs by their own. Some other few comments were made by medical students conveying the need for more pathologies to be added to the detection list of the system.

Interviews

We also conducted interviews as a part of the requirements engineering process of this research. Interviews were explicitly done on a targeted audience to get medical-related knowledge, understanding and the requirements at a concrete level. These were dedicated to the doctors to investigate their requirements on a more profound level. The questionnaire was dedicated to the medical students to capture requirements at a higher level but from a larger audience.

Interviews were conducted in a semi-structured way complying with the questionnaire questions. The selection of the interviewees was made considering their area of expertise. Two doctors with high experience, specifically in thoracic disease diagnosis and one doctor with high experience in the research field were chosen for the interviews. We conducted these interviews via online video conferencing and we spent approximately 20 min on each interview.

Analysis of the interviews done are as below,

Chest radiography is very commonly used, especially in hospitals dedicated to thoracic or other respiratory system-related diseases. Both the doctors mentioned that they take a Chest X-ray (CXR) from almost every patient admitted to the hospital as a standard procedure.

All the interviewees believe that having an automated detection report of the potential findings in a CXR would help proceed with or to confirm the interpreted results from CXR screening further.

All interviewees believe that the system would be even more helpful if it could detect more types of diseases on CXR as numerous types of abnormalities could be present in a CXR.

All interviewees prefer to use the system through a mobile device.

Also, interviewees think that these systems may not be able to replace human radiologists completely.

One interviewee suggested that the accuracy rates must be analytically evaluated, and the error margins must be precisely exhibited in the system interface and in the final detection report to avoid over-dependence on the auto-generated detection report.

ParallelXNet

Why Parallelizing?

Almost all the existing literature uses some sort of an ensembling method in their method. When thoroughly investigating the existing literature related to pathology detection in CXRs, various architectures have been seen to be used for the purpose. An extensive discussion has been driven on these in ("A Concise Review of Existing Work"). Also, our own publication [14] also has some critical discussions. Those discussions show that different neural network architecture performs differently in detecting each different pathology as numerous visually different pathologies could be found using CXRs. Hence our goal was to make the best out of all the architectures.

For the above purpose, a commonly used technique is the Ensembling Method, which basically implies the averaged output of multiple neural networks that have been trained separately. But this method does not guarantee “the best from each” neural network’s that is ensembled together as some outputs might get neutralized in terms of accuracy while the averaging process. For this purpose, we suggest a novel block that can be used, instead of averaging the outputs after the training process.

Parallelize Block

The illustration in Fig. 1 shows the inner connections of the block we propose. This design of the block is mostly motivated by the design behind SE Blocks [34]. As the figure shows, the block has its own mini-neural network embedded into it that learns which channel(s) out of all the neural networks is more important at predicting a given disease. Then this numerical representation of ‘importance’ is used to weight each channel, amplifying the channels that are important. This would highlight the important features out of all the varients of same feature set, identified by different neural networks, which ultimately helps achieving the “best from each” concept.

Fig. 1

An illustration of Parallelize Block’s internal connections

Theory

The Channel Attention based Parallelizing (CAP) block tweaks the contribution of each channel of the final convolutional layer from each of the parallelizing neural network to the final output of the overall model.

Let the inputs for the block be ; where can be varied. and are matched padding with zeroes. Also let are channel counts of the inputs respectively. Then, concatenating the input we get

Also, where represent the nth channel. Then the concatenated input is average pooled to get as below (2)

here will be the input for the embedded fully connected neural network (FCN). FCN is involved in learning the contribution from each channel out of all the channels from all the parallelized neural networks. The FCN behaves as per the below equations.

Let the learning functions of layer FC1 and FC2 be respectively. Then the final output of the FCN, ;where , , and are learned parameters, bias terms, ReLU activation function and the sigmoid function respectively. Note that the dimensionality is reduced by a factor of , which results,

Above represent a vector that learns the significance of each channel that is concatenated in . When properly learned, each element would represent a numerical and quantitative representation for each channel’s ‘importance’ towards the accurate prediction of the final result. To make use of this equation, the vector is then used to weight the channels in . Then the new weighted input is given by (5).where

The new weighted input is again average pooled similarly to Eq. 6 and then connected to the final output dense layer as shown in Fig. 1.

ParallelXNet: Architecture Design

The diagram in Fig. 2 shows the architecture of the ParallelXNet neural network. It contains a common input layer which then connects to the three CNNs. The choice of these three specific architectures was made by investigating their performance, especially in the field of pathology detection in CXRs.

Fig. 2

Overall Architecture of ParallelXNet

We strongly suggest that this can also be used for a variety of applications other than pathology classification in CXRs, which are out of scope for this research. Also, this suggests that various combinations of CNN architectures can be used within this architectural framework proposed depending on the requirement.

Data Preparation

Mainly [11] and [13] datasets were used for this research. Before training, all the datasets were split according to the provided official split. This allowed a fair comparison with all the models and made sure patient overlapping is not present among the splits. As mentioned in previous sections, multiple datasets were used for the development of this system. All the images were resized to 320 × 320 px resolution. The Batch size was set to 16. To avoid unwanted biases and make the image data easy to learn from, sample wise normalization was used in all splits. Data was Augmented in train split by randomly flipping the CXR images while training. The flipping was only done along the horizontal axis. This artificially provides more data for the neural network to train on, ultimately improving the model accuracy.

As no dataset is perfectly balanced, the datasets that we acquired also had this issue of imbalance. Due to this, the loss function will be more affected by negative labels rather than positive labels. Hence, it will make the neural network more biased for the learning of the label type with a higher count when training. To solve this, we used weighted Binary Cross Entropy Loss as the loss function.where is the weighing constant, is the ground truth label and is the predicted label. is the vector containing the frequencies of negative samples per each class. represents the batch size here as we calculate loss per batch.

Training

While training, weights were saved after each epoch. We reduced the learning rate by a factor of 0.1 after training another epoch with the same learning rate when an increase of the validation loss was detected. The minimum learning rate was capped at 0.000001.

Chosen hyperparameters were as per below; these were selected after numerous pre-tests and considering the evidence from the existing literature.

Step per epoch: 4500 [Batch Size: 16]

Learning Rate: 0.0001

Epochs: MIMIC-CXR Dataset-18 | ChestX-ray14 Dataset-9

In summary, for the MIMIC-CXR dataset, we trained the neural network more than 3.5 times, and for the ChestX-Ray14 dataset, 14 times over the whole dataset. Training using the MIMIC-CXR dataset was done in two iterations due to the notebook’s runtime time limit limitations. One Nvidia Tesla P100 16 GB GPU was used in this research for the model fitting purposes.

Web Application

Development Languages

We selected Python [v3.7] as the primary language for the development of the model and the Server of the Web Application. It is an increasingly popular language, especially for Machine Learning and Deep Learning based development work. Python is a language with a very clean and natural syntax yet a very powerful language that allows developers to develop their application following either an object-oriented or a structural approach. The choice of Python is also motivated by its support for the Jupyter Notebooks, which allows the model development to be done more visual and explanatory way. Also, the language supports a variety of frameworks and libraries. These features make the language a perfect entry point for a beginner but who wants to produce something innovative.

Development Frameworks

We used a Python-based framework, Flask [v1.1.2], to develop the back end of the application. The main reason for this choice is the library support it has to the deployment of the deep learning models. The framework has a very easy-to-use simple structure. We firmly believe that Flask is one of the best frameworks to deploy a deep learning model. Flask also provides a very convenient way of developing RESTful APIs based on Python.

Development Architecture

3-Tiered Architecture was selected as a guide in building the system. The functionalities that fulfill the requirements of the system are meticulously placed among the three tiers. Below we discuss briefly about each main tier.

Data Tier

This tier represents the data that is needed for the system to function. The Deep Learning Model, which is the most important component in the system, is placed in this tier. Also, the localized and the uploaded images live in this tier. This saves computational power by eliminating any redundant localizations. The symptoms related to each disease are stored at the server’s side, allowing future changes and improvements.

Logic Tier

This tier contains all the necessary tools need to perform the computations and finally come up with the required output. Sub parts of this tier are as follows.

Detection Engine: Carries out the detection of pathologies on the inputted Chest X-Ray and produces with a probability percentage.

Localization Engine: Localizes the area the potential area that the detected pathologies might exist on the CXR. This is done by generating a heatmap and superimposing it on the original CXR image.

Presentation Engine: Carries out the necessary functionalities needed to present the data generated in the detection engine in an informative way onto the presentation tier.

Presentation Tier

The data generated in the logic tier is processed into a meaningful representation in this tier. These processed results will be then sent to the front end to be displayed to the user when requested. This tier is the interface for the front-end application to the server functionalities.

TensorFlow and Keras

We used TensorFlow-based version of Keras [TensorFlow v2.4.1] to develop the deep learning model. Keras is an easy-to-use, high-level framework that allows developers to build, train and test ML models with a minimum number of code lines. Alternatives like PyTorch was also considered. However, considering the objectives and the requirements of this project, we selected the Keras with TensorFlow backend. Although Keras is a high-level library, its integration with TensorFlow allows the developers to do development at the low level of the library while making the best use of the high-level functionalities. This makes the combination of TensorFlow and Keras beneficial, mostly for a beginner who wants to produce novel and innovative model algorithms.

Evaluation

We consider the evaluation of this system very important, especially as the end system is intended to be used in a medical based environment dealing with humans. Due to this criticality of the system, we first defined our objectives behind the evaluation, which then was used to construct the test goals. The principal testing objectives are as follows.

Certify that the system performs its processes and produces the outputs as expected.

Certify that the ‘algorithm’ and the ‘system’ meet the expected Accuracy, Quality and Standards when performing its processes.

Perceive the reaction and behavior of the system in unexpected situations.

As mentioned earlier, these defined objectives were utilized to define the below test goals.

Goal 1: Investigate the quantitative and qualitative effects of the proposed algorithm.

Goal 2: Investigate the rate of achievement of the defined functional requirements of the system.

Goal 3: Investigate the rate of achievement of the defined non-functional requirements of the system.

Goal 4: Investigate any unexpected behaviors of the system.

In this paper, we focus more on Goal 1 with the aim of portraying the capabilities of the proposed algorithm.

Test Setting

Giving attention to the facts discussed about the accuracy metrics to be unbiased and objective as much as possible, we fixed the below settings for the algorithm.

Parameter Setting

Testing of the model was done extensively under two datasets, as in Table 2. We used one Nvidia Tesla P100 16 GB GPU for our project (see Table 3 for full specification). Only the official splits were used for the tests. That made sure no patient overlapping exists between the splits. For the MIMIC-CXR dataset, we used the NegBio label set due to its better accuracy. For the final model, ‘ParallelXNet-64’ (P-64) and ‘ParallelXNet-128’ (P-128) was used setting and respectively. These were selected after a series of repetitive train-validate cycles with the validation split. We exposed the test split to the model only at the final test inference run, after freezing the hyperparameters. We also suggest that further fine-tuning can be done on these parameters.

Table 2

Dataset split specification

Dataset	Train split	Validation split	Test split
MIMIC-CXR [13]	368,945	2991	5159
ChestX-ray14 [11]	86,524	25,596	[Val. split]

Table 3

Hardware Setting

CPU	Intel xeon
OS	Linux-5.4.89 + -x86_64-with-debian-buster-sid
GPU	Tesla P100-PCIE-16 GB
RAM	16 GB
Hard disk	20 GB

For the MIMIC-CXR, we took weights from the final three epochs after 18 epochs when testing. For the ChestX-Ray14, we trained nine epochs, and the final three’s weights were taken. The six models were then ensembled to get ‘P-Ens’. Test Time Augmentation (TTA) was also used flipping each test sample horizontally. Apart from those mentioned settings, all the other settings were maintained equivalent for the test.

AUC Scores

As shown the Tables 4, 5, Test Time Augmentation has been able to increase the accuracy of the model. Although the increment is not significant, it is consistent throughout the datasets.

Table 4

AUC scores when tested on MIMIC-CXR 2020 dataset

Dataset: MIMIC-CXR 2020
Model pathology	P-64	P-128	P-Ens	P-Ens (with TTA)
Enlarged cardiom	0.7061	0.7076	0.7107	0.7130
Cardiomegaly	0.7921	0.7874	0.7932	0.7939
Lung lesion	0.7155	0.7157	0.7192	0.7218
Lung opacity	0.6978	0.7007	0.7031	0.7037
Edema	0.8403	0.8391	0.8419	0.8427
Consolidation	0.7605	0.7514	0.7597	0.7591
Pneumonia	0.7341	0.7303	0.7372	0.7382
Atelectasis	0.7674	0.7680	0.7703	0.7707
Pneumothorax	0.8595	0.8711	0.8706	0.8704
Pleural effusion	0.8971	0.8952	0.8985	0.8985
Pleural other	0.8313	0.8504	0.8466	0.8448
Fracture	0.6933	0.6810	0.6916	0.6903
Support devices	0.9039	0.9070	0.9085	0.9099

Table 5

AUC scores when tested on ChestX-ray14 dataset

Dataset: chestx-ray14-NIH
Model pathology	P-64	P-128	P-Ens	P-Ens (with TTA)
Nodule	0.7826	0.7807	0.7875	0.7894
Cardiomegaly	0.8901	0.8927	0.8958	0.8967
Emphysema	0.9294	0.9312	0.9335	0.9348
Fibrosis	0.8321	0.8344	0.8381	0.8395
Edema	0.8502	0.8474	0.8526	0.8534
Consolidation	0.7529	0.7527	0.7576	0.7594
Pneumonia	0.7386	0.7353	0.7411	0.7413
Atelectasis	0.7863	0.7823	0.7888	0.7892
Pneumothorax	0.8720	0.8740	0.8773	0.8786
Effusion	0.8359	0.8370	0.8399	0.8406
Mass	0.8329	0.8414	0.8433	0.8435
Infiltration	0.6984	0.7028	0.7041	0.7051
Hernia	0.8742	0.8905	0.8911	0.8928
Pleural thickening	0.7897	0.7889	0.7942	0.7952

Benchmarking

We trained the existing model architectures providing the exact same conditions as in previous tests. Dataset splits and the labels are also the same. The only change done was the model architecture. These conditions and results can be verified by contacting the us, through the model training Jupyter Notebooks we developed. Also see our GitHub repository [https://github.com/SuienS/parallelxnet-cxr-classifier].

Used architectures are,

ResNet50v2 (R-50v2) [30]

DenseNet-121 (D-121) [29]

DenseNet-169 (D-169) [29]

Ensemble of above three, each with the ensemble of the final three epoch weights (R-D-Ens)

The design of the ‘Parallelize Block’ is discussed in ("ParallelXNet"). We introduce this as a better alternative for conventional Ensembling of the Models. This is a solid verification (see Tables 6, 7) because of the consistent results on both the datasets used. Mean values respectively: 0.7885, 0.8246. We strongly suggest this method to be implemented using various architectures available, considering their abilities to detect different pathologies. Also, the number of parallelized architectures can also be increased subjected to the hardware capabilities.

Table 6

AUC scores when tested comparatively on MIMIC-CXR 2020

Dataset: MIMIC-CXR 2020
Model pathology	R-50v2	D-121	D-169	R-D-Ens	P-64	P-128	P-Ens
Enlarged cardiom	0.7026	0.7048	0.7209	0.7159	0.7061	0.7076	0.7107
Cardiomegaly	0.7808	0.7807	0.7888	0.7889	0.7921	0.7874	0.7932
Lung lesion	0.6965	0.7053	0.7111	0.7109	0.7155	0.7157	0.7192
Lung opacity	0.6899	0.6946	0.6967	0.7000	0.6978	0.7007	0.7031
Edema	0.8357	0.8389	0.8434	0.8432	0.8403	0.8391	0.8419
Consolidation	0.7475	0.7507	0.7548	0.7580	0.7605	0.7514	0.7597
Pneumonia	0.7116	0.7228	0.7289	0.7302	0.7341	0.7303	0.7372
Atelectasis	0.7634	0.7627	0.7668	0.7688	0.7674	0.7680	0.7703
Pneumothorax	0.8467	0.8691	0.8640	0.8690	0.8595	0.8711	0.8706
Pleural effusion	0.8897	0.8921	0.8941	0.8957	0.8971	0.8952	0.8985
Pleural other	0.8067	0.8255	0.8544	0.8396	0.8313	0.8504	0.8466
Fracture	0.6613	0.6944	0.6894	0.6891	0.6933	0.6810	0.6916
Support devices	0.8661	0.8994	0.9029	0.9041	0.9039	0.9070	0.9085

‘ParallelXNet’ is better at 9 out of 13 labels of the dataset

Table 7

AUC scores when tested comparatively on ChestX-ray14

Dataset: chestX-ray14-NIH
Model pathology	R-50v2	D-121	D-169	R-D-Ens	P-64	P-128	P-Ens
Nodule	0.7585	0.7736	0.7762	0.7817	0.7826	0.7807	0.7875
Cardiomegaly	0.8770	0.8876	0.8873	0.8943	0.8901	0.8927	0.8958
Emphysema	0.9098	0.9276	0.9259	0.9288	0.9294	0.9312	0.9335
Fibrosis	0.8183	0.8257	0.8359	0.8355	0.8321	0.8344	0.8381
Edema	0.8397	0.8489	0.8471	0.8522	0.8502	0.8474	0.8526
Consolidation	0.7389	0.7443	0.7505	0.7531	0.7529	0.7527	0.7576
Pneumonia	0.7137	0.7287	0.7351	0.7337	0.7386	0.7353	0.7411
Atelectasis	0.7741	0.7799	0.7807	0.7868	0.7863	0.7823	0.7888
Pneumothorax	0.8649	0.8730	0.8733	0.8787	0.8720	0.8740	0.8773
Effusion	0.8248	0.8338	0.8343	0.8376	0.8359	0.8370	0.8399
Mass	0.8128	0.8342	0.8259	0.8361	0.8329	0.8414	0.8433
Infiltration	0.6895	0.7013	0.7031	0.7047	0.6984	0.7028	0.7041
Hernia	0.8703	0.8767	0.8847	0.8880	0.8742	0.8905	0.8911
Pleural thickening	0.7742	0.7899	0.7918	0.7949	0.7897	0.7889	0.7942

‘ParallelXNet’ is better at 11 out of 14 labels of the dataset

Training Performance Testing

Note that for training times in Table 8, hardware is as per Table 3 and training settings and hyperparameter choices are the exact same for all cases. Also, note that the file type that the model is saved is.h5. It saves the whole model in a single file. But further compression mechanisms could be incorporated to reduce model size.

Table 8

AUC scores when tested comparatively on ChestX-ray14

Architecture	Training time (HH:MM:SS)		File size (.h5 type)
	MIMIC-CXR	ChestX-ray14	MIMIC-CXR	ChestX-ray14
ResNet50v2	04:52:21	03:56:40	251 MB	271 MB
DenseNet-121	05:50:16	04:03:29	82 MB	82 MB
DenseNet-169	06:07:08	04:17:37	146 MB	146 MB
ParallelXNet-64	13:37:02	07:18:55	507 MB	507 MB
ParallelXNet-128	13:36:24	07:18:14	503 MB	503 MB

Outcomes from the test show that ResNet50v2 has the shortest training time. Outcomes further show that the DenseNet-121 model has the smallest model size. Although evidently, the ParallelXNet is architecture proven to be better at accuracy, for mobile-based hardware, we recommend using DenseNet-121 due to its significantly low model size. We suggest that ResNet50v2 could be better for re-trainable systems as it has the lowest training time. We also acknowledge that the training time could mainly vary depending on the below key aspects,

Hardware Performance (especially GPU and RAM), ML Library used and Training images’ sizes and the file type.

Generalization Test

The test setting for the generalization test was set to the same as previous algorithm tests. Additionally, we trained ParallelXNet-128 on both MIMIC-CXR and CheXpert datasets. Out of 18 epochs, 9 were trained under MIMIC-CXR, and the latter 9 epochs were trained on CheXpert [12] by Stanford University, US. It contains 224,316 labeled CXRs associated with 65,240 imaging studies. The purpose of this test is to identify how the trained model generalizes between datasets. We note that generalization is mostly affected by the trained set of data and not by the model’s architecture. We also note that although there are 5159 images in the test set of the MIMIC-CXR dataset, only 234 images were given as the test set in CheXpert. This may also have some effect on the test results.

The results in Table 9 show that all the labels other than for one, have good stability with more generalized training. The drastic difference in the ‘Enlarged Cardiom.’ Label is suspected of having caused by the ill performed labeling of the CheXpert labeler, which is also quantitatively shown in [13]. We also emphasize that although some pathologies show an increased AUC, some pathologies have not. This could be due to the difference in the labeling performances, but this also suggests that more data does not always mean more accuracy. Another finding that we could derive from these test results is that the AUC score is not an absolute representation of the accuracy of the model. It represents how well the model can perform on a specific ‘trained distribution’.

Table 9

Generalization test results: ‘Lung Lesion’ results of CheXpert test set was neglected due to insufficient test samples

Train set	MIMIC-2020 + CheXpert		MIMIC-2020
Test set	CheXpert	MIMIC-2020	MIMIC-2020
Model pathology	P-Ens	P-Ens	P-Ens
Enlarged cardiom	0.5361	0.5720	0.7107
Cardiomegaly	0.8149	0.7673	0.7932
Lung lesion	0.3176	0.7201	0.7192
Lung opacity	0.9360	0.6738	0.7031
Edema	0.9173	0.8239	0.8419
Consolidation	0.9299	0.7426	0.7597
Pneumonia	0.6875	0.6864	0.7372
Atelectasis	0.7948	0.7279	0.7703
Pneumothorax	0.8695	0.8881	0.8706
Pleural effusion	0.9337	0.8902	0.8985
Pleural other	0.8927	0.7816	0.8466
Fracture	N/A	0.7190	0.6916
Support devices	0.9650	0.8630	0.9085

We also note that the results are not consistent when captured images from a camera due to various light conditions and camera quality. As also noted in the existing literature, these can be caused by various factors of the camera that is used to capture the CXR. Table 10 shows an example of that. It shows that the images’ technical specifications could significantly affect the inference results of the model even though the images look visually identical to the human eye vision.

Table 10

Localization result on Digital CXR Vs. Camera Captured CXR. The following sample was taken from MIMIC-CXR 2020 [13]

Label: cardiomegaly
Digital CXR	Camera (smartphone) captured CXR
Detection rate: 83.20%	Detection rate: 42.44%

Additional Evaluations on the Model

Furthermore, we tested our model also on the CIFAR-10 dataset to further verify the applicability and performance of it in non-medical scenarios. For the results in Table 11, we only used the official splits for the training and evaluation of the models. Also, note that we used SoftMax activation for the final Dense layer as the data samples are strictly single class. Mathews Correlation Coefficient (MCC) introduced in [43] was also used here to solidify the accuracy results. These results suppose that the parallelization technique that we introduced in this paper can be used instead of typical model ensembling for non-medical use cases as well. ‘R-D-Ens’ and ‘P-Ens’ were trained similarly to it of in ("Training Performance Testing").

Table 11

Overall metrics when tested comparatively CIFAR-10

Dataset: CIFAR-10
Model metric	R-D-Ens	P-Ens
Sensitivity	87.94%	88.55%
Specificity	98.66%	98.75%
Precision	88.00%	88.57%
Accuracy	97.58%	97.72%
Balanced accuracy	93.30%	93.64%
F1-score	87.94%	88.56%
MCC	0.8663	0.8729

‘ParallelXNet’ is better in terms of all the metrics considered for CIFAR-10 dataset

A Critical View on Our Test Results

Parallelization of the CNN Architectures through the introduced ‘Parallelize Block’ performs consistently well across the two datasets used. This proves the effectiveness of the introduced block with solid evidence. The additional test we conducted on the CIFAR-10 dataset further solidifies it. Also, we note that the improvement in the AUC score is limited to 1–2% per disease. However, as the model is intended for medical usage, any improvement could be considered a valuable improvement. Also, our parallelized architecture was evidently seen resilient for different data distributions than others.

Further, we strongly suggest that the introduced parallelized architecture framework can be used with different combinations of CNN architectures which will potentially increase the accuracy scores furthermore.

Limitations and Future Work

Below we acknowledge the predominant limitations of this research.

The models built according to the algorithm introduced have larger file sizes and require high-end hardware to deploy. This might cause scalability issues for a production-level environment.

The built model is only trained on the 14 pathologies which are available in the datasets.

Due to the time restrictions of the Kaggle Notebooks, continuous training was not possible, which led to training the model in parts.

We only tested the accuracy of the localization mechanism qualitatively. Quantitative testing is needed to present further objective test results.

No symptoms or biological signs of the patients were not considered for the detection mechanism, which might increase the accuracy if considered.

We find below features as possible features that could be implemented and integrated into the built system without the need of any significant modifications.

In-system model re-training: This will allow overcoming the issue of generalization. ‘Federated Learning’ based approach could be implemented to enable convenient and effective re-training.

Custom model uploads: This allows users to train models and upload, making the web app universal.

A functionality to allow medical professionals to investigate the patient's medical history directly through the system. This will help them in making diagnosis decisions. Also, this can possibly allow even the model to consider the past CXRs of a patient when his/her new CXR are being detected through the system.

In conclusion, we should note that our work shows a pathway to develop further the existing DL-based models focusing on medical-related applications. Although the models nor the whole system is not recommended for medical usage, our work can be further developed and trained using larger and more varied datasets to increase the accuracy and reliability. We believe systems like these should always be supported and monitored by a trained professional and only be used as a supportive tool. However, we believe more research work like these would one day replace human professionals.

This research is mostly motivated by our passion for machine learning and especially on the advanced mathematics that exists at the core of it. Due to the obscureness of deep learning algorithms we decided to carry out numerous pre-tests covering various aspects of the potential gaps found in the initial literature review. These procedures were done even before the initiation of the project. This helped us to complete the project successfully without significant deviations.