Diagnosis of lymph node metastasis in head and neck squamous cell carcinoma using deep learning.

BACKGROUND: To build an automatic pathological diagnosis model to assess the lymph node metastasis status of head and neck squamous cell carcinoma (HNSCC) based on deep learning algorithms. STUDY
DESIGN: A retrospective study.
METHODS: A diagnostic model integrating two-step deep learning networks was trained to analyze the metastasis status in 85 images of HNSCC lymph nodes. The diagnostic model was tested in a test set of 21 images with metastasis and 29 images without metastasis. All images were scanned from HNSCC lymph node sections stained with hematoxylin-eosin (HE).
RESULTS: In the test set, the overall accuracy, sensitivity, and specificity of the diagnostic model reached 86%, 100%, and 75.9%, respectively.
CONCLUSIONS: Our two-step diagnostic model can be used to automatically assess the status of HNSCC lymph node metastasis with high sensitivity. LEVEL OF EVIDENCE: NA.

INTRODUCTION

Head and neck cancer is the sixth common malignant disease in the world, most of which are squamous cell carcinoma. The prognosis of patients with head and neck squamous cell carcinoma (HNSCC) is poor owing to treatment metastasis, resistance, and recurrence. , Regional cervical lymph node metastasis is an independent poor prognostic factor for patients with HNSCC. , Therefore, accurate diagnosis of lymph nodes metastasis status is an important part of the staging and grading of HNSCC, guiding clinical decision making, and treatment management. In clinical practice, pathological diagnosis is the gold standard for the evaluation of lymph node metastasis, which was estimated by pathologists. However, manual pathological assessment is an exhausting and subjective task. In particular, micro‐metastases (between 2 mm and 0.2 mm) and isolated cancer cell clusters (<0.2 mm) may be ignored and missed. Immunohistochemistry can improve the accuracy of manual evaluation. However, most HNSCC lymph node sections only stained by hematoxylin–eosin (HE) without immunohistochemical detection due to laboratory technology and costs.

Digital pathology includes the process of digitizing histopathological slices and analyzing these digitized images. Digital pathology is widely used in cancer research, including diagnosis, quantitative analysis, biomarkers prediction, and prognostic evaluation. , Deep learning, especially convolutional neural network (CNN), has performed well in the interpretation of digital pathology images. In CAMELYON16 and CAMELYON17 challenges, using artificial intelligence algorithms detected metastases from whole slice images (WSIS) of the HE‐stained breast cancer lymph node, deep learning models even showed similar or better diagnostic performance compared with experienced pathologists. Campanella et al. achieved an area under the curve (AUC) of 0.966 in a dataset of 9894 axillary breast cancer lymph nodes WSIS using weakly supervised learning. Therefore, developing artificial intelligence diagnostic models may be an auxiliary method to solve the shortage of pathologists. However, there is currently no research on artificial intelligence diagnosis of HNSCC lymph node pathological slices.

In this study, we aim to use a deep learning method to extract high‐dimensional features from HE‐stained histopathological images to detect tumor tissues in HNSCC lymph nodes, which finally strives to develop a deep learning‐based model to diagnose the metastasis status of HNSCC lymph nodes and annotate the areas of tumor lesions to assist the pathologist.

MATERIALS AND METHODS

Dataset

A total of 135 HE‐stained lymph node histopathological images from 20 HNSCC patients treated with neck lymph node dissection at Xiangya Hospital of Central South University between August 2018 and May 2019 were enrolled in this study. The digitized histopathological images were acquired at ×20 magnification (0.5 μm/pixel on both horizontal and vertical) by the Pannoramic MIDI scanner (3DHISTECH Ltd, Hungary). The labels of the metastatic status of these lymph node images were based on past pathological reports and manual review. According to the surgery date, 85 lymph node histopathological images from the previous 11 patients were used to construct the development set, and the remaining 50 images were used as the test set (Table 1). The study was approved by the Ethic Committee of the Xiangya Hospital of Central South University (IRB: NO2019121187), while informed consent was obtained from the participants in the study.

TABLE 1

Development set and test set

Dataset	Patients	Positive lymph node images	Negative lymph node images	Total lymph node images
Development set	11	38	47	85
Test set	9	21	29	50
Total	20	59	76	135

CNN‐based primary model

We extracted 40 × 40 pixels patches at 10× magnification (1.0 μm/pixel) as the input unit. All patches containing cancer cells (even if only one cancer cell) were labeled with “tumor”, and the remaining patches were labeled with “nontumor” (Figure 1A). The “nontumor” patches dataset included noncancerous areas from negative lymph node images and positive lymph node images. It should be noted that necrotic tissue and tumor‐associated stroma were also labeled with “nontumor”.

FIGURE 1

Workflows. (A) Training flowchart of the primary model. (B) Training flowchart of the secondary model. (C) Training flowchart of the integrating diagnostic model based on the primary model and the secondary model

We extracted 16,000 “tumor” patches and 16,000 “nontumor” patches from the development set to train the CNN models (Table S1). The patches of each classification were randomly divided into a training set and a validation set at a ratio of 7:3. Four pretrained CNNs based on the ImageNet database (available at http://www.image-net.org/) were trained to classify “tumor” patches from “nontumor” patches: GoogLeNet, MobileNet‐v2, ResNet50, and ResNet101. To increase the generalization and robustness of the model, we augmented the training set by eight times, by rotating (clockwise rotation at 0°, 90°, 180°, and 270°), flipping (in the horizontal or vertical), and combination of these operations. In the training procession, we used the adaptive moment estimation (ADAM) optimizer to update the gradient. The initial learning rate was set to 0.001 and was reduced by a factor of 0.1 after every 10 epochs. The mini‐batch size was 128, and the maximum number of epochs was 30 (the maximum number of training iterations was 42,000). L2 regularization and early stopping mechanism were used to reduce overfitting. The accuracy and the AUC of the receiver operating characteristic curve (ROC) were used to evaluate the classification performance.

Visualization of whole lymph node images based on the primary model

We defined a sliding window with a size of 40 × 40 pixels, nonoverlapping sampling along with the horizontal and vertical directions on the whole lymph node image at 10x magnification. The obtained patches were input into the primary model. If the predicted label was “tumor”, it was marked in red at the corresponding position of the lymph node image, otherwise, it was marked in white. We merged the generated images on the original images to visualize the detection results of the whole lymph node images in the development set by the primary model. The primary model misidentified some negative regions as “tumor”, such as lymphoid follicles and sinus cavities.

CNN‐based secondary model

To reduce false‐positive recognition of the primary model, we analyzed all negative lymph node images in the development set with the primary model. To extract more false‐positive patches, we adopted partial overlapping sampling with 20 pixels as a sliding length (Figure 1B). In this way, a total of 10,137 false‐positive patches were obtained and were labeled with “nontumor”. Then, 10,137 actual “tumor” patches were randomly selected from the primary model dataset and were used to develop the secondary model with those false positive patches. MobileNet‐v2 was used to train the secondary model, which achieved the best performance in the primary classification (Table 2). Similarly, 70% of the image patches were randomly divided into the training set and were augmented, and the rest constituted the validation set (Table S2). The training options and evaluation methods of the second model were roughly the same as those of the primary model.

TABLE 2

Basic characteristics and performance of the four CNNs for classification of the image patches in the validation set

Networks	Depth	Number of parameters (million)	Accuracy (%)	AUC
GoogLeNet	22	7.0	97.3	0.9957
MobileNet‐v2	53	3.5	98.7	0.9982
ResNet50	50	25.6	98.1	0.9974
ResNet101	101	44.6	97.9	0.9975

Abbreviations: AUC, area under the curve; CNN, convolutional neural network.

Lymph node metastasis classification

First, we delineated a tumor probability score heat map of a whole lymph node image. As shown in Figure 1C, a 40 × 40 pixels sliding window continuously cropped image patches in horizontal and vertical directions with steps of 10 pixels. In this way, each pixel, except for the edge area with a width of 30 pixels, was analyzed by 16 times in different patches. Further, the image edge areas (without tissue) were eventually removed. The initial value of the tumor probability score for each pixel was 0. As shown in Figure 1C, a combination of the primary model and the secondary model was used to calculate the tumor probability score for each patch. Specifically, if a patch was recognized as “tumor” by the primary model, then the patch would be entered into the secondary model; otherwise, the patch would be directly labeled as “nontumor”. Furthermore, if a patch was finally recognized as “tumor”, then the tumor probability scores of all pixels located on this patch would be increased by 0.0625; otherwise, the scores would plus 0. Finally, the tumor probability score heat maps of the lymph node images were obtained. Areas with higher scores are more likely to include tumor cells.

Then, we analyzed the distribution characteristics of tumor probability scores of negative and positive lymph nodes in the development set. Finally, the score threshold was set to 0.6875 to classify positive and negative lymph nodes. Images with total scores not less than the threshold were defined as positive; otherwise, the lymph node images were diagnosed as negative. The classification performance of this threshold was tested in the test set. Next, we analyzed the performance of the score threshold in the detection of metastatic lesions.

Statistics

All CNN models were trained on a desktop workstation with a NVIDIA GeForce RTX 2060 super GPU. The statistical analysis and algorithms in this study carried out with MATLAB® (version R2020a). We used the Deep Network Designer app for loading and editing pretrained networks on MATLAB®. On MATLAB, the plotconfusion function was used to plot classification confusion matrix. The perfcurve function was used for plotting ROC curve and calculating AUC. The performance of the primary model and secondary model were evaluated by confusion matrix and AUC. , The classification performance of lymph node metastasis status is evaluated with sensitivity, specificity, and accuracy.

RESULTS

CNNs can distinguish “tumor” patches from “nontumor” patches

The information of patients and lymph node images was shown in Table 1. Four common pretrained CNNs were trained to classify “tumor” patches from “nontumor” patches in the training set (Figure 1A and Table S1): GoogLeNet, MobileNet‐v2, ResNet50, and ResNet101. Then, we evaluated their performance in the validation set. The accuracy of four CNNs all exceed 97%, and the AUCs all exceed 0.99 (Table 2 and Figure S1). These results showed that CNNs could classify “tumor” patches from “nontumor” patches. In particular, the MobileNet‐v2‐based model achieved a classification accuracy of 98.7% and an AUC of 0.9982, both of which were the best among the four models (Table 2 and Figure S1). Hence, we chose the MobileNet‐v2‐based model as the primary model to analyze the whole lymph node histopathological images.

Hard negative regions are incorrectly detected in the whole lymph node images

We next applied the MobileNet‐v2‐based primary model to analyze large‐scale lymph node images in the development set. Reviewing the original images and the visual images, almost all areas of metastasis could be effectively detected. A positive example was presented in Figure 2A. However, we noticed that part of normal lymph node tissues, especially lymphoid follicles and sinus cavities, were misclassified as tumor tissues, as shown in Figure 2B. These misclassifications might lead to misdiagnosis of the lymph nodes. Therefore, we needed to improve the model's classification performance for these normal tissues that were difficult to identify.

FIGURE 2

Visualization of large‐scale lymph node histopathological images analyzed by the primary model. Here are representative examples of correct (A) and incorrect (B) annotation of large‐scale lymph node images in the training set using the primary model with a sliding window of 40 × 40 pixels. The original images are on the left. The merged images are on the right. The area predicted to be tumor is marked in red. The area predicted to be nontumor is marked in white

The secondary model can improve the classification of hard negative patches by the primary model

Misclassifications of lymphoid follicles and sinus cavities were common problems in artificial intelligence detection of lymph node metastases. To improve the classification performance of the diagnostic model for these regions, we constructed a new data set to train the MobileNet‐v2‐based secondary model, consisting of 10,137 misidentified “nontumor” patches and 10,137 randomly matched “tumor” patches (Table S2). The accuracy and AUC of the secondary model in the validation set arrived 96.3% and 0.9933, respectively (Figures 3 and S2).

FIGURE 3

ROC of the secondary model. AUC, area under the curve; ROC, receiver operating characteristic curve

The diagnostic model can exclude approximately three‐quarters of the negative lymph node images while maintaining 100% sensitivity in the test set

Furthermore, we evaluated whether these models could be used to diagnose metastasis status of whole lymph node images. An algorithm was built to make the tumor possibility score heat maps of whole lymph node images using patches‐based classification CNN models (Figure 1C). The distribution characteristics of tumor probability scores in the development set and the test set were shown in Figures 3B and 4A, respectively. In the development set, all positive lymph node images included areas with a score not lower than 0.9375, and the scores of all lymph node negative images were less than 0.6875 (Figure 4A and Table 3). Lymph node images with tumor scores not less than 0.6875 were classified as positive, otherwise, they were classified as negative. We next evaluated the classification performance of the threshold in the test set, as shown in Figure 4B and Table 3, the classification model achieved a sensitivity of 100% and a specificity of 75.9% in the test set.

FIGURE 4

Distribution of tumor probability scores. (A) Distribution of tumor probability scores in the development set. When the score threshold is between 0.6875 and 1, the sensitivity and specificity of the model are both 100%. We set the score threshold to 0.6875 of the lower bound. This means that the area with a score not lower than 0.6875 will be marked as metastatic disease, otherwise, it will be marked as normal tissue. (B) Distribution of tumor probability scores in the test set

TABLE 3

Performance of the integrating diagnostic model

Datasets	Threshold = 0.6875
	Sensitivity	Specificity	Accuracy
Development set	38/38 (100%)	47/47 (100%)	85/85 (100%)
Test set	21/21 (100%)	22/29 (75.9%)	43/50 (86%)

Next, we analyzed the similarity between the area of the lymph node image where the tumor score was not lower than the threshold and the true metastatic disease area. Our results indicated that the model annotated the main metastatic lesions and excluded most of the nontumor areas in the development set and the test set (Figure 5). Interestingly, cancer cell clusters smaller than 200 microns were also occasionally detected (Figure 5A). However, there were still six false‐positive lymph node images and local false‐negative areas for pixel‐level metastasis detection (Figure S3). Nevertheless, no positive lymph node images were classified as negative.

FIGURE 5

Representative examples of the delineation of metastatic lesions by the diagnostic model. The areas detected as metastatic lesions are marked in red. The area detected as normal tissue is marked in white. (A) The metastatic lesions with accurate segmentation in the development set. (B) A correctly classified negative lymph node in the development set. (C) The metastatic lesions with accurate segmentation in the test set. (D) A correctly classified negative lymph node in the test set

DISCUSSION

Previous studies have reported the application of artificial intelligence to detect lymph node metastasis in breast cancer, , , lung cancer, colorectal cancer, and so on. Recently, there are some radiomics studies to predict HNSCC lymph node metastasis or extranodal extension from lymph node computed tomography (CT) imaging. However, the disadvantage is that only large enough lymph nodes can be detected from CT. A study predicts the presence or absence of lymph node metastasis from early‐stage oral tongue squamous cell carcinoma clinical and primary tumor pathology records, but it is impossible to evaluate specific individual lymph nodes. In this pilot study, we developed a CNN‐based HNSCC lymph node diagnosis model and tested the classification performance on a dataset based on the images with or without metastasis. Our data indicate that this model can diagnose all positive lymph node images and exclude approximately three‐quarters of negative lymph node images. To our knowledge, this is the first application of deep learning from cervical lymph nodes HE‐stained pathological images in the prediction of lymph node metastasis in patients with HNSCC.

One advantage of this study is that we developed a secondary model to reduce the false positive recognition of hard negative regions. The primary model based on MobileNet‐v2 achieved an accuracy of 98.7% and an AUC of 0.9982 in the patches classification in the validation set; however, when analyzing in large‐scale images, some hard negative areas were identified as metastatic lesions, mainly located in lymphoid follicles and sinus cavities. This is a common phenomenon in the development of artificial intelligence models for lymph nodes in other malignancies, , which may be caused by a low proportion of hard negative patches in the training set. Therefore, researchers often used hard negative mining to increase the proportion of these hard negative patches in the training set. , Previous study in lung cancer has trained an additional model to eliminate lymphoid follicles in lymph node to reduce the false positive detection, however, the hard negative areas apart from the lymphoid follicles were ignored in this model. In our study, we re‐collected these misclassified patches to train a secondary model to analyze the patches defined by the primary model as “tumor”, to improve the classification performance between tumor and hard negative regions.

Common postprocessing methods in the lymph node‐based classification include training new machine learning models and setting lesion size thresholds based on the generated probability heat maps. , In a previous study, the specificity of the artificial intelligence detection model without postprocessing was even 0. However, postprocessing might cause false‐negative results. In the CAMELYON17 Challenge, the best model even classified 4 macro‐metastasis (larger than 2 mm) WSIS as negative. A previous study set the threshold of positive lesion size to 0.6 mm and 0.7 mm, and the final sensitivity was 79.6% and 75.5%, respectively. In clinical practice, it is often unacceptable for the missed diagnosis of positive lymph nodes in malignancies. In our study, we calculated the tumor score by partially overlapping sampling and selected the score threshold to diagnose the status of lymph node metastasis based on the difference between the tumor score in positive and negative images. Our model can screen out a large number of negative lymph nodes with a sensitivity of 100% in our test set, freeing the pathologist to focus on the diagnosis of difficult slices. Meanwhile, as for lymph nodes that are diagnosed as positive by the model, pathologists also need to be cautious to distinguish malignant lesions from false‐positive identifications.

Next, we analyzed the role of the score threshold in annotating metastatic lesions. In many cases, the annotated lesions were basically consistent with the actual lesions, and even some tumor cell clusters smaller than 0.1 mm were also marked. In addition, the model identified the most negative lymph node images. We have to mention that some tumor tissues were ignored by the score threshold. As shown in Figure S3A,B, these metastatic lesions were not completely marked. Nevertheless, these false‐negative phenomena did not lead to the missing of diagnosis in our test set. Increasing the number of patches that include only a few tumor cells in the training set may improve the performance of the model. These results indicate that the threshold in the pixel‐wise detection for metastatic lesions still needs to be improved and it is necessary to further validate the performance of the model in the lymph node slices with micro‐metastases and isolated cancer cell clusters.

Focusing on the false‐positive lymph node images in the test set, one of these was predicted with a large area of metastasis (Figure S3C). We noticed that the staining of this lymph node was weaker than most other lymph node images. Color differences are rare in our dataset but cannot be ignored in slices from different institutions. To reduce the interference caused by color differences, common image preprocessing methods including color normalization and color augmentation are required. , , Besides, the rest of the false‐positive lymph node images were marked with only a few scattered tumor spots, mainly located in lymphoid follicles and sinus cavities (Figure S3D).

There are some limitations in our study. Our training samples size is limited, and collected from one institution. A large‐scale multi‐institutional study in future will improve the robustness and generalizability of the model. Another limitation is that this is a retrospective study, and further prospective studies are necessary. Notwithstanding these limitations, this pilot study suggests that it is possible to develop a CNN‐based automatic diagnosis model of HNSCC lymph node metastasis with clinical applications.

CONCLUSIONS

In summary, our results suggest that it is possible to automatically detect metastasis from HNSCC lymph node pathological images. In our study, we developed and evaluated a two‐step deep learning model to screen suspected positive HNSCC lymph nodes and outline the regions of interest to assist pathologists. This method reduces false‐positive identification and maintains high sensitivity, but it still cannot replace manual evaluation. Whether this method can be applied to complex clinical practice will still require the further evaluation.

CONFLICT OF INTEREST

The authors have declared no conflicts of interest for this article.

FIGURE S1 Confusion matrices of four CNN pretrained models in the primary model validation set. After training, we evaluated the performance of the four models on the validation set of 4800 “tumor” patches and 4800 “nontumor” patches. (A–D) respectively showed the confusion matrices of GoogLeNet, MobileNet‐v2, ResNet50, and ResNet101 for classifying “tumor” patches from “nontumor” patches.

Click here for additional data file.

FIGURE S2 Confusion matrix of the secondary model. Classification of “tumor” patches from “nontumor” patches in the secondary model validation set with 3041 “tumor” patches and 3041 hard “nontumor” patches.

Click here for additional data file.

FIGURE S3 Analysis of the false‐negative areas and the false negative images by the diagnostic model. (A) The elongated lesions almost ignored in the development set. The dotted area pointed by the arrow is the actual tumor lesion. (B) The metastasic lesions almost ignored in the test set. The dotted area pointed by the arrow is the actual tumor lesion. (C) A negative lymph node with large false‐positive lesions in the test set. (D) A false‐positive lesion less than 0.1 mm, located in a lymphoid follicle of a negative lymph node in the test set.

Click here for additional data file.

TABLE S1 Dataset of primary model.

TABLE S2. Dataset of secondary model.

Click here for additional data file.