An Open Medical Platform to Share Source Code and Various Pre-Trained Weights for Models to Use in Deep Learning Research.

Deep learning-based applications have great potential to enhance the quality of medical services. The power of deep learning depends on open databases and innovation. Radiologists can act as important mediators between deep learning and medicine by simultaneously playing pioneering and gatekeeping roles. The application of deep learning technology in medicine is sometimes restricted by ethical or legal issues, including patient privacy and confidentiality, data ownership, and limitations in patient agreement. In this paper, we present an open platform, MI2RLNet, for sharing source code and various pre-trained weights for models to use in downstream tasks, including education, application, and transfer learning, to encourage deep learning research in radiology. In addition, we describe how to use this open platform in the GitHub environment. Our source code and models may contribute to further deep learning research in radiology, which may facilitate applications in medicine and healthcare, especially in medical imaging, in the near future. All code is available at https://github.com/mi2rl/MI2RLNet.

INTRODUCTION

Deep learning is one of the most sought-after artificial intelligence (AI) research area in the medical field [1]. Studies across multiple medical specialties have employed AI to mimic the diagnostic abilities at or above the level of human experts [23456]. To develop these AI algorithms to perform tasks typically associated with human intelligence, one dataset of labeled cases with which the AI algorithms are trained and another dataset with which they are tested or validated are required. However, thus far, they have not been easily accessible to AI developers because they often contain sensitive patient information that would compromise patient confidentiality and privacy [7].

Prior to its expansion into the medical field, deep learning has been revolutionized through the use of open databases and innovation. Image databases, such as MNIST [8], ImageNet [9], COCO [10], and CelebFaces [11], have been used not only as datasets for the development of deep learning models but also as sources of open code, networks, and pre-trained weights. This open innovation approach made it easy for research groups around the world to share their experiences, leading to the increased popularity of deep learning approaches.

Although it is well known that transfer learning from a pre-trained model that was trained using ImageNet could be helpful when researchers use a small dataset in radiology [12], radiology-related open-source code and open trained models are still limited. In addition, constructing medical big data is difficult for local hospitals. To tackle this problem, in this study, we released various types of pre-trained weights for various models through an open platform, MI2RLNet, for downstream tasks. We hope that our released model will have a positive effect on individual transfer learning. Figure 1 shows the overall architecture of MI2RLNet. We provide detailed examples for classification, detection, and segmentation tasks, and summarize how to use the open platform in the GitHub environment.

Fig. 1

Overall architecture of MI2RLNet.

LR = left/right

Description of MI2RLNet

In this paper, we introduce MI2RLNet, which consists of nine models that derive results for a specific purpose using medical images of various body parts obtained using different modalities. To train the MI2RLNet models, most data were obtained from open datasets, and some anonymized data were obtained from the Asan Medical Center (AMC). All AMC datasets were approved by the Institutional Review Board of AMC. The imaging data were anonymized in accordance with the Health Insurance Portability and Accountability Act Privacy Rule. Detailed information regarding our data is presented in Tables 1 and 2.

Table 1

Task Description of MI2RLNet with Models’ Performance

Task	Dataset	Modality	Part	Model	Performance
Enhanced/non-enhanced classification	AMC	CT	Chest	ResNet-50 [15]	Acc: 0.99
L/R mark detection	AMC	X-ray	Chest	EfficientDet [16]	mAP: 0.99
Lung segmentation	JSRT [18]	X-ray	Chest	2D U-Net [13]	DSC: 0.98
Kidney & tumor segmentation	KiTS19 [19]	CT	Abdomen	3D U-Net [14]	DSC: 0.83
Brain extraction tool	AMC, OASIS [20]	MRI	Brain	2D U-Net [13]	DSC: 0.95

KiTS19 (University of Minnesota Medical Center) [19], Liver Tumor Segmentation Challenge, OASIS [20]; For detection, Mean Average Precision at intersection over union 0.5, For segmentation. Acc = classification accuracy, AMC = Asan Medical Center, DSC = dice similarity coefficient, JSRT = Japanese Society of Radiological Technology, KiTS19 = Kidney Tumor Segmentation Challenge, L/R = left/right, mAP = mean average precision, OASIS = Open Access Series of Imaging Studies

Table 2

Dataset Description of MI2RLNet

Task	Unit	Dataset	Total	Train	Validation	Test
Enhanced/non-enhanced classification	Slices (patients)	AMC	74978 (200)	59982 (140)	7498 (40)	7498 (20)
L/R mark detection	Patients	AMC	10411	6767	1822	1822
Lung segmentation	Patients	JSRT [18]	247	150	50	47
Kidney & tumor segmentation	Patients	KiTS19 [19]	300	168	42	90
Brain extraction tool	Patients	AMC	57	40	8	9
		OASIS [20]	70	54	8	8

AMC = Asan Medical Center, JSRT = Japanese Society of Radiological Technology, KiTS19 = Kidney Tumor Segmentation Challenge, L/R = left/right, OASIS = Open Access Series of Imaging Studies

In this paper, we briefly introduce our representative five models that may be widely used; the other four models not specifically addressed in this article include view classification in chest X-ray, liver segmentation in abdominal CT, polyp detection in endoscopy, and black-blood segmentation on MRI. To confirm the rationality of our implementation, we used network architectures that are commonly used in the computer vision field, such as U-Net [13], three-dimensional (3D) U-Net [14], ResNet [15], EfficientDet [16], and EfficientNet [17]. The performance is listed in Table 1, and the details of each task, such as hyperparameters and training environments, are summarized in the Supplement.

Classification of Enhanced and Non-Enhanced CT Images

In actual clinical settings, there can be many errors in the digital imaging and communications in medicine (DICOM) header, especially when the images are collected from multiple centers, including incorrect recording of enhanced and non-enhanced CT imaging acquisition, which could strongly affect the accuracy of deep learning models. Therefore, we developed a classifier that consists of ResNet-50 [15] as a backbone network to classify enhanced and non-enhanced chest CT images. As a result, the model showed a 0.99 accuracy with the test set.

Detection of Left/Right (L/R) Mark on Chest Radiographs

An left/right (L/R) mark, typically seen as a strong white letter object, is located on every X-ray image, and it is not related to the status of the patient. However, the training of neural networks is usually affected substantially by pixels that have a strong intensity on images. If the L/R mark is erased, the network will be able to learn better representative features. Therefore, we developed an L/R mark detection model, which consists of EfficientDet [16] with EfficientNet-B0 [17] as a backbone network, to detect and erase the mark. As a result, the model showed a 0.99 mean average precision with the test set. Figure 2 shows the results of the L/R mark-detection model.

Fig. 2

Results of the left/right mark detection model (A-C).

Lung Segmentation in Chest Radiographs

The aim of chest radiography is to identify abnormalities in the lung region; therefore, obtaining a lung mask for chest X-ray images is very important. In addition, this procedure can be used as a pre-processing step for the registration, focusing, or reduction of false positives in irrelevant regions. In particular, registration of the field of views of given images based on the lung region on chest X-ray images can be used to overcome differences in the pose and breath-hold level of each lung. Therefore, we developed a lung segmentation model, which consists of U-Net [13], to obtain lung masks. As a result, the model showed a 0.98 dice similarity coefficient (DSC) with the test set. Figure 3 shows the results of the lung segmentation model.

Fig. 3

Results of the lung segmentation model.

A. Input. B. Ground truth. C. Predicted result.

Kidney and Tumor Segmentation

Obtaining kidney and tumor masks is very important because of the wide variety in their morphology and how their morphology relates to surgical planning and outcomes. Therefore, we developed a kidney and tumor segmentation model, which consists of a cascaded 3D U-Net [14], to obtain kidney and tumor masks. Figure 4 shows the cascaded 3D U-Net architecture. As a result, the model recorded the 33rd place in KiTS19 [13] challenge with a total DSC of 0.83 (0.96 for the kidney and 0.70 for the tumor) with the test set. Figure 5 shows the results of the kidney and tumor segmentation models.

Fig. 4

The overall architecture of the kidney and tumor segmentation.

A. Cascaded 3D U-Net with SE-block for segmentation of the kidney and tumor on abdominal CT. First network detects ROIs in the entire image and second network segments detailed labels within the ROIs. B. 3D U-Net architecture with SE-block. ReLU = rectified linear unit, ROIs = regions of interest, SE-block = Squeeze-and-Excitation block, 3D = three-dimensional

Fig. 5

Results of the kidney and tumor segmentation model.

A. Ground truth. B. Predicted result.

Brain Extraction Tool for MRI

Brain extraction is a fundamental step in brain analyses. To perform an effective analysis, brain skull stripping is usually performed, but manual skull stripping is time-consuming. Therefore, we developed a fully automated segmentation model, which consists of U-Net [13], to automatically extract the brain area on T1-weighted MRI. As a result, the model showed a DSC of 0.95 with the test set. Figure 6 shows the results of the brain extraction model.

Fig. 6

Results of the brain extraction model for different diseases.

How To Use MI2RLNet

In this section, we briefly describe how to use MI2RLNet. One module, liver segmentation, is presented with pseudocode as an example; other specific instructions can be found in the README in the GitHub repository. We also describe how MI2RLNet can be fine-tuned to fit custom datasets. The performance of the modules of MI2RLNet is excellent; however, for new data, the performance may be lower than expected. In this case, MI2RLNet can be fine-tuned for the new dataset to achieve robust performance for extended tasks. For more detailed information on each model, please refer to the Supplement, which includes typical models for classification, detection, and segmentation in chest X-ray, CT, and MRI, respectively. This paper can be cited as a reference when using models in future research.

Inference

MI2RLNet is implemented so that users can use it easily. Furthermore, each module is well-modularized; thus, the desired results can be obtained with only a few lines of code. To use this concise library, only three steps need to be taken: initialize the module, set the model with pre-trained weights, and run. The “Checker” object is also provided, which can check the data type for each module and set up resources, such as Graphical Processing Units. Although the modules in MI2RLNet are implemented in different deep learning frameworks, such as TensorFlow [21] and PyTorch [22], we provide an integrated environment independent of these frameworks as required.

The guideline for this simple inference is introduced in Algorithm 1, with liver segmentation as an example. Detailed instructions on how to use other modules can be found in the README of each module in the GitHub repository.

Algorithm 1. Guideline for the inference of MI2RLNet

# 1. Initialize

# Import the module you want to use from medimodule.Liver import LiverSegmentation

# 2. Set the model with weight model = LiverSegmentation(“path/of/weight”)

# 3. Run

# Get a result of the image

# ‘save_path’ must be set to save the result in the module image, mask = model.predict(“path/of/image”, save_ path=”path/for/save”)

Fine-Tuning

Although we trained our model using big datasets for robustness, the accuracy may be affected by the CT, X-ray, and MR devices used. Different standards between hospitals can also affect the model accuracy. Fine-tuning the released model weights can stabilize the training of the model from the beginning, and the accuracy can be increased for existing models. Detailed instructions on how to transfer this model into a more fine-tuned model can be found in the README of each module in the GitHub repository. Algorithm 2 shows an example of the fine-tuning of the liver segmentation model.

Algorithm 2. Guideline for the fine-tuning of MI2RLNet

# 1. Initialize

# Import the module you want to fine-tune from medimodule.Liver import LiverSegmentation

# 2. Set the model with its weight for fine-tuning

# If you want to train random initialized model,

# you don’t have to set the weight model = LiverSegmentation(“path/of/weight”)

# 3. Run

# Construct your custom training code

...

model.train()

...

DISCUSSION

Deep learning is becoming increasingly important in the medical field and may enable accurate predictions from the analysis of complex medical data. Deep learning requires sufficient training datasets and algorithms to optimize the performance on the training dataset before testing. Large datasets are required to achieve sufficient accuracy, and the data must be appropriately labeled. The difficulties in collecting data from medical records are as follows: access to relevant medical records may be difficult, particularly if treatment is spread over multiple different providers; time-consuming administrative process to acquire access to medical records; heavy burden of filtering the data to meet research goals; and the accuracy of some data collection systems may be questionable. Owing to these difficulties, deep learning-based studies have mainly been conducted in large hospitals.

Deep learning technologies related to medical data organization should be widely shared for the scalability of deep learning-based research in the medical field. We released pre-trained deep learning models that can be applied in various medical deep learning studies. This could help researchers to start their own deep learning research with fewer obstacles. For example, chest radiographs may have multiple views, such as posterior-anterior and lateral views, consequently increasing the difficulty of training the deep learning model. One of the models of MI2RLNet that we shared here could potentially be useful after fine-tuning in other hospitals.

To develop a deep learning solution that guarantees generalizability, medical images should be processed considering the characteristics of the modality. We trained models for various tasks with various modalities through the classification, detection, and segmentation of X-ray, CT, and MR images. The pre-processing code we introduced could help colleagues start new tasks, and if they perform transfer learning using the trained models, they could start with smaller datasets or train their model faster.

As we trained the models using public datasets and datasets from AMC (non-public datasets), our trained models may not be able to cover all possible sources of variability. In the future, we will keep our open platform sustainable by expanding its application to new tasks, covering more datasets, and considering the feedback of colleagues worldwide.