Emerging Applications of Deep Learning in Bone Tumors: Current Advances and Challenges.

Deep learning is a subfield of state-of-the-art artificial intelligence (AI) technology, and multiple deep learning-based AI models have been applied to musculoskeletal diseases. Deep learning has shown the capability to assist clinical diagnosis and prognosis prediction in a spectrum of musculoskeletal disorders, including fracture detection, cartilage and spinal lesions identification, and osteoarthritis severity assessment. Meanwhile, deep learning has also been extensively explored in diverse tumors such as prostate, breast, and lung cancers. Recently, the application of deep learning emerges in bone tumors. A growing number of deep learning models have demonstrated good performance in detection, segmentation, classification, volume calculation, grading, and assessment of tumor necrosis rate in primary and metastatic bone tumors based on both radiological (such as X-ray, CT, MRI, SPECT) and pathological images, implicating a potential for diagnosis assistance and prognosis prediction of deep learning in bone tumors. In this review, we first summarized the workflows of deep learning methods in medical images and the current applications of deep learning-based AI for diagnosis and prognosis prediction in bone tumors. Moreover, the current challenges in the implementation of the deep learning method and future perspectives in this field were extensively discussed.

1 Introduction

Bone tumors occur in the musculoskeletal system and can be divided into primary and metastatic bone tumors. It is reported that the incidence of primary bone tumors is 2–3 per 100,000 people, accounting for about 6.2% of all tumors (1–4). Primary bone tumors can also be subdivided into benign, intermediate, and malignant tumors. Among them, osteosarcoma is the most prevalent primary malignant bone tumor and contributes to the second cause of death in adolescents and children (1, 5–8). Notably, without timely diagnosis and treatment, patients with a malignant bone tumor may suffer the risk of a worse prognosis, such as amputation or metastasis (2, 9, 10). Therefore, early detection and treatment are pivotal for limb salvage and reducing morbidity and mortality. Currently, the conventional diagnosis procedure for a bone tumor is a combination of clinical characteristics, imaging, and pathological examinations since bone tumors are rare. However, it is not easy for clinicians to achieve accurate and timely diagnoses since the available diagnostic process can be laborious, high cost, time consuming, and could be biased by clinicians’ expertise and experience (11, 12). Accordingly, there is an urgent need for a more rapid, reliable, and accurate diagnosis of bone tumors in clinical practice.

In the field of healthcare, artificial intelligence (AI) has a wide range of applications, including imaging and diagnostics, lifestyle management and supervision, nursing, emergency and hospital management, drug mining, virtual assistants, wearables, and more (13, 14). AI in the healthcare industry can tremendously improve the efficiency of clinical work and reduce the shortage of medical resources. Since deep learning technology began to be applied to image recognition tasks in 2012, their recognition performance has reached a high point in recent years. The research methods have been widely used in medical image analysis and processing tasks. Also, the applications of the deep learning method in medical images provide assistance for disease diagnosis automatically with shortened time, enhanced efficacy, and favorable accuracy (15).

Due to the advancement in AI, techniques based on deep learning such as segmentation, detection, classification, and enhancement have been successfully applied to the field of medical imaging (15–26), bringing new opportunities for building computer-aided medical imaging diagnosis system. In recent years, deep learning methods have been successfully used in musculoskeletal imaging for lesion identification and severity assessment, such as a fracture (27–36), knee lesion (37–42), osteoarthritis (43–47), and spinal degenerative lesion (48, 49). In addition, some models based on deep learning are adopted to assess bone age (50–55) and determine sex (56) on radiographs. Therefore, it is plausible to use deep learning methods to establish diagnostic models for bone tumors as well, which may greatly reduce the misdiagnosis and missed diagnosis rates of bone tumors. Recently, a growing number of studies have reported that deep learning-based AI models in bone tumor identification, classification, segmentation, and visual interpretation could improve diagnostic, prognostic, and predictive accuracy, demonstrating its great potential application in bone tumors. Moreover, often compared with deep learning, radiomics is also an advanced technology that is often cooperated with artificial intelligence, designed to extract, and analyze numerical radiological patterns based on quantitative image features, including geometry, size, texture, and intensity. It is well known that radiomics could be applied in disease prediction, prognosis, and monitoring (57, 58). The purpose of this review is to briefly describe the concepts and workflows of deep learning-based AI, extensively summarizing its up-to-date applications in bone tumors, as well as discuss the barriers to deep learning implementation and future directions in this field.

2 Artificial intelligence based on deep learning

AI can refer to a branch of computer science that can simulate human intelligence. AI is implemented in machines to perform tasks that require human knowledge in an automated manner (59). On the other hand, machine learning (ML) is a subset of AI and is defined as mathematical algorithms that enable a machine to make choices independently without any external human influence (60). Furthermore, deep learning, as a novel approach, is a subset of ML, but there are differences between ML and deep learning in data dependencies, hardware dependencies, feature engineering, problem-solving approach, execution time, and interpretability (61). Deep learning can learn to perform much more complex classification tasks from input such as images, text, or sounds, achieving superior performance to traditional machine learning (62). Accordingly, deep learning models contain many layers to build neural network architectures and need to be trained by a large set of labeled data.

In 1980, Fukushima (63) developed the neocognitron architecture to automatically identify the best features of an image, which is thought to be the prototype of a convolutional neural network (CNN). Convolution filters contained in CNNs can automatically extract features, while models with simple convolution layers can extract low-level visual information such as color, texture, and shape, and deeper convolution layers can extract abstract semantic information. Until 2012, a deep convolution network called AlexNet, proposed by Krizhevsky, had excellent performance in the ImageNet competition, providing a powerful method far surpassing the traditional method of manually designing features (64). Since then, the deep learning method has gradually become a hot topic in AI research. In 2017, Mishra et al. first introduced deep learning models (AlexNet, LeNet, and VGGNet) to improve the efficiency and accuracy of differentiating osteosarcoma tumor tissues and their counterparts (65). Since then, a substantial number of studies focused on deep learning in bone tumors have emerged in recent years. The relationships, development, and comparison of AI, ML, deep learning, radiomic, and deep learning in bone tumors are presented in detail in , .

Figure 1

Timeline of the development of artificial intelligence, machine learning, radiomics, deep learning, and application of deep learning in field of bone tumor.

Figure 2

Brief comparison of the pipeline of radiomics, conventional machine learning, and deep learning.

3 Workflow for building a deep learning model

Deep learning technology enables the process of automatic feature extraction by learning deep features and relationship patterns of images directly from the architecture of multiple network layers on the original data (or preprocessed slightly) instead of manual extraction in the conventional ML method. In this way, the efficiency of image analysis is greatly improved without time-consuming manual feature extraction, and the system based on deep learning technology provides assistance for those without professional experience.

Building a typical deep learning model for medical image analysis consists of three sections ( ): (1) preprocessing images for sufficient quantity and suitable quality annotated data and splitting the dataset with an appropriate proportion (often 70%:20%:10% for training, validation and test dataset); (2) training the deep learning model on the training dataset and refining parameters; and (3) evaluating the performance of the model under the test dataset.

Figure 3

Workflow of building a deep learning model for bone tumor classification based on radiological and pathological images, including preprocessing, training, and evaluation.

3.1 Image Preprocessing

Image preprocessing is a significant process to obtain a training dataset for building a deep learning model with high-quality and uniform format images. Since images may are collected from different instruments and through distinct acquisition settings, standardization of images can maintain data consistency and comparability for the same format as input. The detection network is utilized to show the area of interest on the image with a bounding box for segmentation, facilitating image mining and analysis. If there are not enough training images, image augmentation methods like digitally reconstructed radiographs (66–68) and generative adversarial networks (GANs) (69, 70) can be used to enlarge the image dataset. Remarkably, image labeling is the most time-consuming work because it requires researchers to manually annotate input–output pairs based on their clinical experience, but it is also the most critical step since these annotations are considered as “Gold Standard” for training and evaluation. The images obtained by the abovementioned methods are more suitable for model training than the original images and can significantly improve the performance of the model.

3.2 Model Training

CNNs are the most prevalently used multilayer neural network structures in image-based deep learning, and various CNN architectures like VGG, ResNet, and DenseNet have been used in medical image-based models. A typical CNN model consists of convolution layers, pooling layers, activation functions, and full-connected layers. In addition to batch standardization, dropout is also applied to optimize the performance of the model. Typically, in deep learning models based on CNN, there are three steps in data processing by the convolution layer. First, the convolution layer obtains a set of linear transformation outputs by multiple convolution calculations, and then each linear output is processed by a nonlinear activation function as a probe, and finally, the pooling function is used to further adjust the output of the convolution layer. It is important to note that the CNN model needs to be trained on a large amount of dataset for refining parameters. If the number of data is not enough, the model will overfit on the trained dataset and cannot be generalized in real data and will have poor performance.

3.3 Model Performance Evaluation

It is important to prepare a test dataset that has the same distribution as the training dataset. Running the trained model on the test dataset can provide an objective evaluation of model performance. There are many common evaluation metrics, including accuracy, precision, recall, sensitivity, specificity, the area under the curve (AUC) of the receiver operating characteristic (ROC) curve, and F1-score. Sometimes, we use single-number evaluation metrics like accuracy to simply compare different models, but most of the time, we utilize various metrics to evaluate models for a wide variety of uses. If we have to take both accuracy and running time into account, we can first define an “acceptable” running time, typically less than 100ms, and then maximize the classifier’s accuracy as much as possible within a limited running time frame. At this point, the running time represents the “satisfaction indicator,” and accuracy means the “optimization indicator.”

4 Deep learning applications in medical images for bone tumors

In the published studies of deep learning, multiple models were developed for analyzing radiological and pathological images, showing excellent performances that can be comparable to those of experienced physicians (including orthopedic surgeons, radiologists, and pathologists). Among them, the most analyzed radiological images for deep learning are generated from X-ray, CT, and MRI (71–80). These radiological images are used in deep learning for tumor detection and classification; differentiation of benign, intermediate, and malignant tumors; segmentation of the region of tumors; and tumor grading prediction. In addition, bone scintigraphy, PET, and spectral CT are also good tools to detect bone metastasis (81–93), as evidenced by their ability to identify the primary lesion of the tumor and calculate the volume of the metastatic sites, and applying deep learning to these radiological images can improve our diagnosis of bone metastasis. With regard to the pathological images, it is demonstrated that these features from resected tissues can reveal tumor histopathology after hematoxylin and eosin (H&E) staining and can also be used for tumor classification, prognosis prediction, and treatment guidance. Various models have been proposed to help clinicians diagnose and identify tumor regions on digital H&E-stained tissue images (65, 94–97). The details of the application of deep learning in primary and metastatic bone tumors are depicted in and presented as follows.

Figure 4

A scheme showing the process of incorporating deep learning models to assist bone tumor diagnosis and thereby facilitate decision making in clinical practice.

4.1 Deep Learning in Primary Bone Tumors Based on Radiological Images

4.1.1 Lesion Detection and Classification

X-ray, CT, and MR images are mainly used in deep learning for lesion detection and classification. Generated by X-ray, plain radiographs contain image parameters of tumor location, tumor size, and the margin of tumor for describing tumor characteristics. In 2020, He et al. (73) first developed a deep learning model based on CNN that could automatically classify primary bone tumors based on 2,899 plain radiographs from 1,356 patients using a multi-institutional dataset. The model had a high performance with an AUC reaching up to 0.877 and 0.916 for classifying benign and malignant, respectively, and with an accuracy of 72.1% for three-way classification (benign vs. intermediate vs. malignant), which was closed to 2 subspecialists, and outperformed junior radiologists. Similarly, combining global context and local patch for analysis, Do et al. (76) built a Multi-Level Seg-Unet model for the detection and classification of knee bone tumors on plain radiographs using 1,576 radiographs consisting of 1195 tumor images and 381 normal images with a superb accuracy of 99.05%.

Compared with plain radiographs, CT and MRI can provide further radiological information and improve lesion detection. Multiple deep learning methods have also been published for detecting and classifying bone tumors on CT and MRI (75, 77). For instance, a deep learning-based radiomics model (75) was described for discriminating between benign and malignant sacral tumors using 3D CT and clinical characteristics based on 1,316 manual-cropped radiomics features from 459 patients and achieved a high AUC of 0.83 in identifying benign and malignant sacral tumors. MRI is highly sensitive for the detection of bone abnormalities due to its ability to evaluate bone marrow involvement, soft tissue invasion, and fluid content of lesions (98, 99). A deep learning model (77) based on routine MRI and patient demographics using the EfficientNet-B0 architecture and a logistic regression model has also been described for identifying benign and malignant bone lesions, with an expert-level performance of accuracy, sensitivity, specificity, and AUC of 0.76, 0.79, 0.75, and 0.82, respectively.

4.1.2 Segmentation and Volume Calculation

Another potential application of deep learning and radiological image analysis is the automatic segmentation and volume calculation of tumors (71, 72). Due to the large spatial and structural variabilities of tumors, osteosarcoma tumor segmentation on CT has been a challenging difficulty in AI. In 2017, Huang et al. (71) developed a multiple supervised fully convolutional networks (MSFCN) method to segment the region of tumors automatically. The deep end-to-end model used multiple feature channels to capture more context information and achieved an average DSC of 87.80%, an average sensitivity of 86.88%, an average HM of 19.81%, and an F1-measure of 0.908. A similar multiple supervised residual network (MSRN) model (72) based on ResNet cooperated with FCN and also has a high performance for osteosarcoma segmentation trained and tested on 1,900 CT images from 15 osteosarcoma patients. Furthermore, using a multiview fusion network to extract pseudo-3D information, a deep learning model (80) had the ability to perform the segmentation and volume calculation of pelvic bone tumors in MRI and reduce the average segmentation time by 100 times compared to other methods, which has a significant impact on the clinical practice of musculoskeletal oncology.

4.1.3 Tumor Grading

Tumor grading is crucial for treatment plan making and prognosis prediction (100). A deep learning-based tumor grading model on T1-weighted or T2-weighted MRI sequences was proposed by Navarro et al. (79) in soft-tissue sarcoma patients. Grading the sample into low grade (G1) and high grade (G2/G3), the deep learning method achieved an F1-score of 0.90 and an AUC of 0.76. It provides insight for surgeons to make treatment plans while reducing invasive biopsies. Although there is currently no similar research in primary bone tumors, it may be a potential research direction in the near future.

4.1.4 Tumor Necrosis Rate Assessment

Evaluation of tumor necrosis rate after neoadjuvant chemotherapy in patients with malignant bone tumors is critical since it can show the treatment response of patients and thus offer guidance for the subsequent chemotherapy after surgery (101). Currently, this process is assessed by multiple pathological slices, and it is so complex and time consuming that it limits its clinical practicality. Thus, it is necessary to convey other convenient and noninvasive methods to identify the tumor necrosis caused by neoadjuvant chemotherapy. To a great extent, it may facilitate our clinical evaluation of tumor chemotherapy sensitivity and help us classify patients into responders (≥90% tumor) and nonresponders (<90% necrosis) (102).

In 2020, a preliminary study conducted by Huang et al. applied ML to predict tumor necrosis rate on multiparametric MRI before and after chemotherapy in patients with osteosarcoma (103). This study is of great significance since it first explored the potential correlation between contrast-enhanced MRI and postoperative pathological features. More recently, other research performed by Kim et al. (104) first demonstrated that texture features of ML of positron emission tomography/computed tomography (PET/CT) images could reflect the fluorine-18fluorodeoxyglucose (18F-FDG) uptake heterogeneity features in osteosarcoma and therefore predict the treatment response to neoadjuvant chemotherapy. Although these studies used a supervised ML method instead of deep learning due to the small sample size, it is plausible and meaningful to expand the sample size and apply deep learning to predict histologic response by adopting other radiological imaging modalities (MRI, PET/CT), which may help clinicians to decide whether to continue the prior chemotherapy regimen to treat patients after surgery (104).

4.1.5 Prognosis Prediction

The recurrent risk of bone tumors after surgery is a matter of great concern to orthopedic surgeons (105). Treatment for a bone tumor like curettage may be less invasive than en bloc wide resection. Meanwhile, the rates of recurrence and metastases are increased (2, 106). Thus, it is crucial to predict postsurgery regional recurrence of tumors based on presurgery medical images, which is not easy for physicians. Based on 56 patients with confirmed giant cell bone tumors (GCTB) in histopathology after curettage, a CNN model was reported by He et al. (107) to predict the local recurrence of GCTB on presurgery MRI. Without integrating any clinical characteristics, the pure image-based CNN model achieved an accuracy of 75.5% and a sensitivity of 85.7%. Using logistic regression, the fusion model was built by integrating tumor location, patient age, and CNN prediction, and the accuracy and sensitivity for prediction were improved to 78.6% and 87.5%, respectively, much higher than that of the radiologists (64.3% and 58.3%) (107). It is obvious that the regression deep learning model combined with clinical characteristics has a great potential to predict tumor recurrence and thereby guide clinical decision making.

4.2 Deep Learning in Bone Metastasis Based on Radiological Images

In recent years, various technologies have been used in the diagnosis of bone metastasis from different primary tumors, including bone scintigraphy, CT, MRI, and PET/CT (108, 109). The deep learning method is utilized in these modalities to detect the presence of bone metastasis, segment and calculate the volume of the metastatic lesions, differentiate the source of the primary tumors, and denoise scintigraphy to improve the quality of images.

Bone scintigraphy with 99mTc-MDP is widely applied in the detection and localization of bone metastasis in cancer patients since it has the merit of whole-body detection and high sensitivity for the diagnosis of bone metastasis (110–112). Several studies (86–90, 92, 93, 113–115) have explored applications of deep learning models in interpreting images of bone scintigraphy, and these models aid in diagnosing lesions and reducing the workload for clinical physicians. Papandrianos et al., (88, 89) proposed two deep learning models based on RGB-CNN architecture that can be trained efficiently on a small dataset and consume less running time to identify the bone metastasis and differentiate between a bone metastatic and a degenerative lesion in prostate cancer patients. Despite the models using small datasets (778 patients for model 1 and 507 patients for model 2) without any clinical information as input, the classification accuracy for two classes (bone metastasis or healthy) and three classes (normal, malignant, and degenerative) reached up to 97.38% and 91.42%, respectively, outperforming than other well-known CNN approaches. Cheng et al., (93) Cheng et al., (113) proposed CNN models to detect bone metastasis in the pelvis and identify metastasis spots on the ribs or spine in the chest from prostate cancer patients’ bone scintigraphy, with a sensitivity of 0.87 by using a hard positive mining (HPM) approach as an effective augmentation method and faster R-CNN and YOLO v3 to identify hotspots. Compared with analyzing only one image at a time, Pi et al. (90) developed a novel CNN approach to analyze both the anterior and posterior views of WBS examination for the presence of bone metastasis in various cancer patients and use a spatial attention feature aggregation operator for better spatial location information. Trained on 15,474 examinations from 13,811 patients, the model had excellent performance with an F1, accuracy, sensitivity, and specificity of 0.933, 95.00%, 93.17%, and 96.60%, respectively. Similarly, a multi-input CNN model (92) was designed for identifying bone metastasis in breast, prostate, lung, and other cancer patients based on a large number of bone scans (12,222 cases of bone scintigraphy). Using multiple images as input, the machine performed well, with the AUC of 0.98, 0.955, 0.957, and 0.971 for breast cancer, prostate cancer, lung cancer, and other cancers, respectively. It is noted that integrating local and global information would improve the performance of the model, Han et al. (114) proposed a 2D whole body-based and “global-local unified emphasis” (GLUE) CNN model for detecting bone metastasis on bone scans with a high accuracy of 0.900 and 0.889. In addition, some studies (86) utilized deep learning to denoise scintillation camera images, significantly promoting the ability of the model to detect bone metastasis in whole body bone scans by a Monte Carlo simulation approach.

Moreover, multiple deep learning methods have been used to identify bone metastasis on CT and MRI as well. In 2018, Chmelik et al. (82) reported a CNN-based method to classify and segment spine metastasis of lytic and sclerotic lesions in whole-spine CT scans with 1,046 lytic lesions and 1,135 sclerotic lesions from 31 cases. Trained on voxel-wise labeled images by two independent radiologists, the machine had 92% sensitivity in classifying and localizing small metastatic lesions greater than 1.4 mm3 under object-wise evaluation. Similarly, a deep learning-based DC-U-Net model (116) was also built to identify and segment spinal metastasis in lung cancer on spectral CT (dual-energy) images and have expert-level performance. Fan et al. (117) proposed a deep approach that employed the AdaBoost algorithm to classify images and the Chan-Vese (CV) algorithm to segment the lesion for the diagnosis of spinal bone metastasis in lung cancer patients on MRI images, achieving a high classification accuracy of 96.55%. Also, a deep learning model (115) using 2,880 annotated CT scans of 114 patients with prostate cancer was able to detect and classify bone lesions as benign and malignant, with a high accuracy of 92.2%. Volume calculation of metastatic lesions is important for clinical decision making. Lindgren Belal et al. (84) trained a CNN approach on 100 CT scans that was capable of segmenting and calculating the volume of the metastatic lesions authomatically with performance comparable to that of an experienced radiologist.

In addition to detecting bone metastasis, a deep learning model can also be applied to identify the origin of metastatic lesions. A DCE-MRI analysis method (83) based on deep learning was developed by Lang et al. (83) to detect bone metastasis in the spine and distinguish the tumor originating from primary lung cancer or other cancers. Compared with the methods based on hotspots or radiomics, the deep learning model had better performance for differentiating the origin of the metastatic lesions with an accuracy of 0.81, which would assist in predicting the primary cancer source when expansive PET/CT is not available.

4.3 Deep Learning in Bone Tumors Based on Pathological Images

Pathological images based on formalin-fixed paraffin-embedded (FFPE) tissues are often used as the gold standard for routine diagnosis of bone tumors. Tumor histopathology revealed by HE-stained tissue images and immunohistochemistry (IHC) for specific biomarkers can provide crucial information for the prediction of their clinical prognosis and sensitivity to therapy. In recent years, deep learning methods have been used to diagnose and identify tumor regions on digital H&E-stained tissue images as the reference standard.

To classify the region of viable and necrotic tumor in whole slide images (WSIs) of osteosarcoma, trained on 536 nontumor tiles, 263 necrotic tumor tiles, and 345 viable tumor tiles annotated by two pathologists, a CNN model was proposed as a classifier for differentiating osteosarcoma WSIs into a viable tumor, necrotic tumor, and nontumor in H&E-stained images with an accuracy of 91.2% (94). Tumor-prediction maps were utilized to visualize the tumor region and type classified by the CNN model, while they can also be used for calculating the percentage of the necrotic region and showing the area of the tumor over the WSIs. A similar CNN model was demonstrated by Mishra et al. with an accuracy of 92.4%, a precision of 97%, and an F1-score of 95%. By using the Siamese network and a new residual block called DRB, a DS-Net classifier composed of ASN and CN can utilize paired data as the input and refine the architecture through input pairs labeled by experts (95), and the model reached up to an average accuracy of 95.1% for the classification of viable and necrotic tumor regions in osteosarcoma. Moreover, a CNN classifier utilized by Foersch et al. (97) can also distinguish subtypes of soft tissue sarcoma from histopathological slides as well, and the accuracy of the pathologists was markedly improved from 46.3% to 87.1%.

In addition to tumor diagnosis assistance, the deep learning model cooperated with the application of a statistical approach for survival analysis and can also provide prognosis prediction of sarcoma (97). In leiomyosarcoma, CNN classifiers with the use of H&E-stained tissue images were shown to be useful for survival analysis with a mean AUC of 0.91 and an accuracy of 88.9%. In addition, class activation maps were used to visualize image regions and features that the model detected, showing that the prediction regions of “dead” were associated with fewer lymphoid infiltrates, more prominent intercellular matrix, presence of intratumoral hemorrhage, and more tumor-associated vessels, which are known as valuable prognostic factors for tumors (97).

5 Remaining limitations and future perspectives

Despite the application of various successful deep learning algorithms (details in ), several limitations and potential perspectives need to be addressed as well.

Table 1

Summary of applications of deep learning-based artificial intelligence in bone tumors.

Authors (year)	Input feature	Applications	Deep learning methods	Size of dataset				Performance
				Dataset	Training	Validation	Testing
He et al. (118)	X-ray	Classify primary bone tumors	CNN	2,899 images/1,356 patients	70%	10%	20%	Accuracy: 0.734AUC: 0.877 (benign); 0.916 (malignant)
Do et al. (76)	X-ray	Detect, classify, and segment knee bone tumors	Multilevel Seg-Unet model	1,576 images	80%	20%	NA	Accuracy: 0.99Average mean IoU: 0.848
Liu et al. (78)	X-ray; clinical characteristics	Classify benign, intermediate, and malignant tumors	CNN (Inception_v3)	982 images	784 images	97 images	101 images	AUC: 0.898 (benign); 0.894 (malignant); 0.865 (intermediate)Macroaverage AUC: 0.872
Huang et al. (71)	CT	Segment osteosarcoma	CNN (VGG-16); multiple supervised side output layers (MFSCN)	2,305 images/23 patients	1,900 images	NA	405 images	Sensitivity: 86.88%F1: 0.908DSC: 87.80%Average HM: 19.81%
Yin et al. (75)	CT; clinical characteristics	Classify benign or malignant sacral tumors	DNN	1,316 images/459 patients	321 patients	138 patients	NA	Accuracy: 0.81AUC: 0.84
Eweje et al. (77)	MRI; clinical characteristics	Classify benign and malignant bone lesions	CNN (EfficientNet)	1,060 images	70%	20%	10%	Accuracy: 0.76Sensitivity: 0.79Specificity: 0.75AUC: 0.82
Papandrianos et al. (89)	Bone scintigraphy images	Identify the presence of bone metastasis in prostate cancer	CNN	778 patients	505 patients	156 patients	117 patients	Accuracy: 0.9142
Zhao et al. (92)	Bone scintigraphy images	Identify bone metastasis	CNN	12,222 patients	9,776 patients	1,223 patients	1,223 patients	AUC: 0.988 (breast cancer);.955 (prostate cancer);.957 (lung cancer);.971 (other cancers)
Papandrianos et al. (88)	Bone scintigraphy imaging	Classify malignant (bone metastasis) or healthy in prostate cancer	CNN	586 images	68%	17%	15%	Accuracy: 0.9738Sensitivity: 0.965Specificity: 0.968Precision: 0.969Recall: 0.974F1: 0.97
Cheng et al. (93, 113)	Bone scintigraphy images	Identify bone metastases in the pelvis, ribs, or spinal cord	R-CNN; CNN (YOLO v3)	576 WBBS images: 205 prostate cancer/371 breast cancer	NA	NA	NA	Sensitivity: 0.82 for chest; 0.87 for pelvisSpecificity: 0.81 for the pelvisPrecision: 0.7 for the chest
Han et al. (114)	Bone scintigraphy images	Detect bone metastasis in prostate cancer	CNN	9,133 bone scans/5,342 patients	Abundant: 72%Limited: 10%	Abundant: 8%Limited: 40%	Abundant: 20%Limited: 50%	Accuracy: GLUE: 0.900; WB: 0.889AUC: GLUE vs. WB: 0.894–0.908 vs. 0.870–0.877
Pi et al. (90)	Bone scintigraphy images	Identify bone metastasis	CNN; SDNN	15,474 images/13,811 patients	12,274 images	1,600 images	1,600 images	Accuracy: 0.95Sensitivity: 0.9317Specificity: 0.961MSE: 0.933
Lang et al. (83)	DCE-MRI	Differentiate spinal metastases originating from lung and other cancers	Convolutional long short-term memory (CLSTM) network	61 patients	NA	NA	NA	Accuracy: 0.810
Masoudi et al. (115)	CT	Classify benign or malignant bone lesions in prostate cancer	CNN (2D ResNet-50; 3D ResNet-18)	2,880 CT scans/114 patients	75%	12%	13%	Accuracy: 92.2%F1: 92.3%
Fu et al. (95)	H&E slides	Classify viable and necrotic tumor regions in osteosarcoma	CNN (DS-Net)	1,144 images	60% (654)	20% (218)	20% (219)	Accuracy: 0.951Sensitivity: 0.920Specificity: 0.961Precision: 0.929F1: 0.922
Mishra et al. (65)	H&E slides	Classify tumor (viable tumor, necrosis) and nontumor region in osteosarcoma	CNN	82 WSIs/1,000 images	60%	20%	20%	Accuracy: 0.924Precision: 0.970Recall: 0.940F1: 0.950
Arunachalam et al. (94)	H&E slides	Classify viable and necrotic tumor regions in osteosarcoma	CNN	40 WSIs/1,144 tiles	NA	NA	NA	Accuracy: 0.912

AI, artificial intelligence; AUC, area under the curve; CNN, Convolutional Neural Networks; CT, computed tomography; H&E, hematoxylin and eosin; MRI, magnetic resonance imaging; NA, not available; SDNN, Standard Deep Neural Network; WBBS, whole-body bone scan; WSI, whole slide images.

Firstly, training a deep learning model needs substantial labeled data. While image collection and annotation are the most time-consuming tasks, they need an experienced physician for image annotation since this labeling would be the gold standard for later training and testing. The quality of training datasets determines the model’s performance. However, it is pretty difficult to achieve in a bone tumor, which may be attributed to the following reasons: (1) the incidence and prevalence of bone tumors, such as osteosarcoma, are pretty low, thus the number of cases is comparatively small, and most of the data are concentrated in large teaching hospitals; (2) it is expensive and sometimes unrealistic to require experienced physicians to spend much time in labeling images with high quality; (3) medical images involve patient privacy, and in the ethically demanding medical field, it is difficult to obtain the permission to use of large-scale data.

Secondly, there has been significant progress in data collection and sharing in recent years. A number of public datasets like TCGA (119), CCC-EMN, and Cancer Genome Atlas Research Network (120) for cancers have been published, and several methods using public datasets achieved good performances. Unfortunately, data concerning bone tumors are absent in most available public databases, creating obstacles to conducting deep learning in this field. However, transfer learning refers to training models on other abundant medical images and transferring them to target images, which may be a good tool since the amount of data and required time will be greatly minimized when learning new tasks (121, 122).

Thirdly, most published models were designed to deal with a single task. However, in clinical practice, physicians do not rely on a single medical file for a final diagnosis. For instance, physicians need to incorporate the clinical parameters and radiological and pathological images to achieve the diagnosis of osteosarcoma. Furthermore, in pathological diagnosis, the pathologists also need to take into account the results of other IHC biomarkers, such as Ki-67, SATB2, and MDM2, or even next-generation sequencing (NGS), to reach a confirmed diagnosis in addition to H&E staining (123–126). If the models are built based on a single parameter, their clinical value may be greatly jeopardized. Therefore, more comprehensive models combining various characteristics should be designed and emphasized in the future.

Fourthly, primary bone tumors and tumor bone metastases were mentioned above, while studies regarding lung metastasis of malignant bone tumors, the main cause of death in sarcoma patients, have not been explored yet. Early identification of lung metastasis is critical for alleviating prognosis and treatment of patients (127). Recently, deep learning approaches have been applied to detect and classify lung metastasis and build deep learning-based image reconstruction techniques (128, 129) that can significantly reduce the radiation dose of CT (103). Due to the crucial role of lung metastasis detection for prognosis in malignant bone tumors, more efficient models are expected to be developed.

Fifthly, the genetic information of tumors can be used to predict tumor subtypes and the prognosis of patients, but the related profiling remains costly. In recent years, gene expression prediction based on deep learning models has become a research hotspot with the development of high-throughput sequencing as well as the application of deep learning in genomics and transcriptomics. Several models based on deep learning in gene expression prediction have shown good performance in several tumors (125), such as lung cancer (130–132) and breast cancer (73, 133). However, the relevant studies have not been performed on a bone tumor, and it has gained a promising expectation to apply deep learning for the prediction of gene expression in the bone tumor.

Lastly, spectroscopy, such as Raman spectroscopy, can provide quantifiable and label-free information for molecular patterns of diseases and can be combined with deep learning to identify and diagnose tumors, which is a novel and promising field. Currently, there are several models using spectroscopy to predict several types of tumors, such as pancreatic cancer (134), nasopharyngeal cancer (135), breast cancer (118, 136), lung cancer (137), and tongue cancer (138). However, the combination of spectroscopy and deep learning is scarce. Only one CNN model (91) based on 1,281 serum Raman spectra from 427 patients was reported to identify prostate cancer bone metastases with high performance, and there is no related study in primary bone tumors. Therefore, it is promising to develop deep learning approaches based on spectroscopy with or without radiological/pathological images to pave a new avenue for the identification of bone tumors.

6 Conclusion

In conclusion, there have been various studies using deep learning methods in bone tumor diagnosis on radiological and pathological images. Many of them have shown excellent performances in detection, classification, segmentation, and volume calculation of the primary tumor and bone metastasis. In addition, some models taking other clinical characteristics into consideration were proposed to predict the prognosis of cancers with even higher accuracy, and the prediction of the deep learning model was demonstrated as an independent prognosis factor. However, there are few models that have been really applied in clinical workflow, which may be attributed to their poor generalization capability to clinical practice. Currently, the suitable training data are insufficient and most models can only process a single data, opposing barriers to its implementation. Future research may deal with these limitations and focus on a more diverse way to implement deep learning models into clinical practice. Generally, deep learning methods could provide powerful assistance for clinicians and reduce time consumption and economic burden while optimizing clinical treatment strategies in bone tumors.

Author Contributions

XZ: investigation, data analysis, and visualization, writing—original draft preparation. HW, CF, RX, YH, and LL: investigation and editing. CT: conceptualization, supervision, writing, and revision. All authors have read and agreed to the published version of the manuscript.

Funding

This work was supported by the National Natural Foundation of China (81902745), Hunan Provincial Natural Science Foundation of China (2022JJ30843), the Science and Technology Development Fund Guided by Central Government (2021Szvup169), Hunan Provincial Administration of Traditional Chinese Medicine Project (No. D2022117), and the Clinical Research Center for Medical Imaging in Hunan Province (2020SK4001).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.