Automated detection of vascular remodeling in tumor-draining lymph nodes by the deep-learning tool HEV-finder.

Vascular remodeling is common in human cancer and has potential as future biomarkers for prediction of disease progression and tumor immunity status. It can also affect metastatic sites, including the tumor-draining lymph nodes (TDLNs). Dilation of the high endothelial venules (HEVs) within TDLNs has been observed in several types of cancer. We recently demonstrated that it is a premetastatic effect that can be linked to tumor invasiveness in breast cancer. Manual visual assessment of changes in vascular morphology is a tedious and difficult task, limiting high-throughput analysis. Here we present a fully automated approach for detection and classification of HEV dilation. By using 12,524 manually classified HEVs, we trained a deep-learning model and created a graphical user interface for visualization of the results. The tool, named the HEV-finder, selectively analyses HEV dilation in specific regions of the lymph nodes. We evaluated the HEV-finder's ability to detect and classify HEV dilation in different types of breast cancer compared to manual annotations. Our results constitute a successful example of large-scale, fully automated, and user-independent, image-based quantitative assessment of vascular remodeling in human pathology and lay the ground for future exploration of HEV dilation in TDLNs as a biomarker.

Introduction

Dilation of the specialized blood vessels in lymph nodes, the high endothelial venules (HEVs), has been reported in several tumor types, including squamous cell carcinoma of the tongue and oral and pharyngeal carcinoma [1, 2], and may be a common cancer‐induced dysregulation of the lymph node (LN) functions in cancer. We have demonstrated that dilation of HEVs can be seen in high frequency in tumor‐draining (TD) LNs from patients with invasive ductal carcinoma (IDC) [3]. These changes, in contrast, are rare in the noninvasive breast cancer subtype ductal carcinoma in situ (DCIS). Thus, HEV dilation in tumor‐draining lymph nodes (TDLNs) is associated with tumor invasiveness in estrogen receptor positive (ER+) breast cancer. The TDLNs have recently gained new attention as potential sites for therapeutical targeting in cancer, especially in the context of immunotherapy [4, 5] and it is well established that TDLNs play a central role for development of tumor immunity [6]. LN metastasis is common in several types of human cancer, often associated with worse prognosis, and is the basis for more aggressive treatment in e.g. breast cancer [7]. Currently, in clinical practice LNs are only analyzed for the presence of metastasis. To improve prediction of risk for distant metastasis and in the context of immunotherapy to improve assessment of tumor immunity status, additional biomarker analysis of TDLNs has gained new interest [4].Vascular changes in TDLNs have prospects as future biomarkers, alone or in combination with other biomarkers in oncoimmunology. However, current solutions for in situ multiplex image analysis of the immune contexture in human tumors focus on single‐cell annotations and cell counts [8, 9]. Image analysis tools to assess specific vascular structural changes within the tissue are largely missing. In this article, we address this need.

Artificial intelligence (AI)‐driven solutions, and more specifically deep learning, for image analysis have been shown to be highly effective for both supervised and unsupervised recognition of tissue patterns in general pathological stained sections, and have great potential to improve accuracy, reproducibility, and speed of medical diagnostics, and to ease workloads for clinicians, as reviewed by van der Laak et al [10]. We present here a deep‐learning‐based model for automated detection and classification of HEVs in multicolor immunofluorescence‐stained sections. We compare the automated HEV detection and classification by HEV‐finder to manual/visual analysis of the same regions of interest (ROIs) in patients with ER+ breast cancer and validate the performance of the tool to independently identify patients with a high degree of HEV‐dilation from our previously published dataset [3]. To further evaluate the HEV‐finder, we applied it for the analysis of a new cohort of breast cancer patients with triple negative (TN) breast cancer. Our data support that HEV dilation is common in both ER+ and TN breast cancer. In summary, our data provide a successful example of how to approach automated analysis of vascular changes in human cancer and provide a validated tool for assessment of HEV dysregulation in TDLNs.

Materials and methods

Biobank material and ethical considerations

A previously published collection of formalin‐fixed and paraffin‐embedded (FFPE) human biobank LNs collected at Uppsala Biobank between 2004 and 2019 were used for machine learning [3]. The cohort included patients with the noninvasive breast cancer DCIS (n = 19 patients) and the hormone‐sensitive (ER+, PR+/−, HER2−, low/int Ki67 [from 5 to >50%]) invasive IDC with (n = 17 patients) or without (n = 22 patients) LN metastasis. Three patients per group were used for the validation analysis and were excluded from the training by deep learning. An additional cohort for application of the HEV‐finder included patients with TN IDC (ER−, PR−, HER2−, 48–95% Ki67), with (n = 8 patients) and without (n = 7 patients) LN metastasis (supplementary material, Table S1). Both cohorts included only patients with no neoadjuvant treatment and no distant metastasis at the time of surgery. One random section per patient was used for analysis. Data on disease recurrence and distant metastasis can be found in Bekkhus et al [3] for the hormone‐sensitive cohort and in supplementary material, Table S1 for the TN cohort.

The study was conducted in accordance with the Helsinki Declaration of 1975, revised in 1983, and was approved by the regional Ethical Committee in Uppsala, Sweden. Approval: 2017/061 (03 May 2017) and addition 2017/061:1 (07 February 2018) and 2017/061:2 (17 July 2018) to Maria H. Ulvmar. Patient consent was waived, since only archival biobank tissue samples were used and anonymous clinical data.

Immunofluorescence staining

HEVs were defined by immunofluorescence staining for the HEV marker peripheral node addressin (PNAd) together with the vascular marker Claudin‐5 and were analyzed in the paracortex (T‐cell zone) of the LNs based on the presence of the marker CCL21 [3]. Metastatic tumor cells were defined by the expression of cytokeratin [3]. The staining of FFPE LNs was previously described by Bekkhus et al [3]. In brief, 4‐μm thick sections were deparaffinized and rehydrated by incubation in xylene, then an ethanol gradient. Antigens were retrieved by incubation in 1 mm ethylenediaminetetraacetic acid (EDTA, Invitrogen, Waltham, MA, USA; pH 9) at 97 °C. Endogenous biotin and avidin was blocked by incubation in 0.033% of streptavidin (Sigma Aldrich, St. Louis, MO, USA) and 0.0033% of biotin (Sigma Aldrich) followed by blocking in 5% donkey serum (Sigma Aldrich). All blocking reagents were diluted in wash buffer (PBST; PBS with 0.05% Tween20 [Sigma Aldrich]). Primary antibodies (listed in supplementary material, Table S2) were diluted in PBST with 5% donkey serum and incubated overnight at 4 °C. Secondary antibodies (listed in supplementary material, Table S3) and CF647 conjugated streptavidin (Biotium, Fremont, CA, USA) were diluted in PBST only and incubated for 30 min at room temperature (RT) separately. All sections were counterstained with 4’,6‐diamidino‐2‐phenylindole (DAPI) (Invitrogen). Sections were mounted using ProLong Gold antifade mounting media (Invitrogen) and #1.5 coverslips. Images were acquired using a Vectra PolarisAutomated Quantitative Pathology Imaging System (Akoya Biosciences, Menlo Park, CA, USA). A 20× objective (5 μm/pixel), whole‐slide scan with filers for DAPI, Opal520, Opal570, Opal620, and Opal 690 as described [3].

Automated detection of vascular remodeling

To create and evaluate the HEV‐finder tool for automatic analysis of HEV dilation in human biobank tissues, we used our published dataset from our manual image analysis of axillary LNs from patients with DCIS and ER+ IDC with and without LN metastasis [3]. The manual/visual assessment of HEV dilation started by defining tiles within the paracortex (T‐cell zone) of the LNs, defined by the presence of the marker CCL21 [3]. HEVs close to adipocytes or to metastatic tumor cells (i.e. <200 μm) were excluded. HEVs were visually classified as nondilated, intermediately dilated (lumen < 10 μm), or highly dilated (lumen > 10 μm) (Figure 2A). HEV dilation status was saved to a spreadsheet. Our fully automated detection and classification of vascular remodeling, HEV‐finder, strives to mimic this manual/visual process. It is initiated by automatically creating a mask for the CCL21 positive area (i.e. paracortex) and placing tiles randomly within this area. The mask for the CCL21 positive area is found using a local variance‐based threshold. Next, a Mask R‐CNN [11] neural network using the Keras [12], available at: https://github.com/fchollet/keras, and Tensor Flow [13] libraries, trained on 12,524 manually classified HEVs from 51 patients, detects and classifies all HEVs within every tile. HEVs close to adipocytes and metastatic tumor cells were excluded. This was achieved by defining adipocyte regions as dark tissue regions (i.e. with intensity lower than a fixed threshold) with a contour curvature higher than a fixed threshold, and flagging HEVs closer than 200 μm to an adipocyte region. Metastatic tumor cells are defined by an intensity threshold applied to the cytokeratin channel, and all detected HEVs closer than 200 μm to the resulting region are flagged (Figure 3A). Finally, HEV locations, dilation status, and potential flags are saved to a spreadsheet. HEV‐finder also exports an image with tiles and outlines of HEVs, adipocyte regions, and tumor regions. The number of tiles the HEV‐finder defines depends on the LN size and the area of the paracortex. The image‐processing steps of the HEV‐finder are in Python, using the numpy and scikit‐image libraries (https://numpy.org/ and https://scikit-image.org/) directly on the raw data.

Figure 2

Validation of detection and classification of HEVs using HEV‐finder: a deep‐learning‐based tool. (A) Immunofluorescence (IF) staining of peripheral node addressin (PNAd) (magenta), Claudin‐5 (cyan) and DAPI (blue) to visualize examples of nondilated (non‐dil), intermediately dilated (int‐dil), and highly dilated (high‐dil) HEVs. Scale bar, 20 μm. (B) ROIs of HEVs (PNAd+/Claudin‐5+, magenta/cyan) showing both manual and automatic detection and classification of non‐dil (red), int‐dil (green), and high‐dil (blue). Scale bar, 20 μm. (C) Confusion matrix of the manual versus automatic classification of lymph nodes (LNs) (n = 3 sections per group), not used for training, draining the noninvasive breast cancer ductal carcinoma in situ (DCIS), and the invasive ductal carcinoma with (IDCmet+) and without (IDCmet−) LN metastasis.

Figure 3

Comparison of manual analysis and the HEV‐finder for identification of patients with HEV remodeling in ER‐positive breast cancer. (A) Example of the QuPath view with selection of ROIs for analysis using HEV‐finder. Lymph node (LN) compartments visualized by immunofluorescence (IF) staining of cytokeratin marking the metastatic cells (red), CCL21 (green) for the paracortical fibroblasts, peripheral node addressin (PNAd) (magenta) for the high endothelial venules (HEVs) and Claudin‐5 (cyan) for the HEVs and lymphatic endothelium. Cytokeratin mask annotated in gray, CCL21 area in magenta, and non‐dil, int‐dil, and high‐dil in red, green, and blue, respectively. Scale bar: 250 μm. Image with annotations only can be found in supplementary material, Figure S1. (B) Zoom‐in on the left tile in panel (A). (C) Dotplot visualizing the accuracy of the AI‐driven analysis to identify differences between patient groups: noninvasive ductal carcinoma in situ (DCIS), invasive ductal carcinoma, without IDCmet− and with IDCmet+ metastasis, in the degree of HEV remodeling. The same LNs but different tiles, and thus vessels, are analyzed in the manual and the automatic analysis. The three IDC patients with the highest percentage of highly dilated HEVs are marked by separate symbols. Nonparametric Kruskal–Wallis test and Dunn's multiple comparisons test were used for statistical analysis. *p < 0.05, **p < 0.01, ***p < 0.001.

Evaluation of the HEV‐finder

Automated and manual HEV detection and classification was compared by manually analyzing 10 randomly selected tiles per LN (n = 9 LNs not used for training) containing a total of 1791 HEV ROIs detected and classified by HEV‐finder. The confusion matrixes presented in Figure 2C compare the automated classification to the manual HEV detection. As a validation of the accuracy for the AI‐driven analysis to identify differences between patient groups, 10 patients with the lowest percentage, defined by the manual analysis, were selected from the DCIS group and the 10 patients with the highest percentage were selected from the two IDC groups. A collection of 15 TN breast cancer LNs was included to apply the HEV‐finder in an independent sample group and different tumor type. Eight samples with LN metastasis (TNBCmet+) and seven samples without LN metastasis (TNBCmet−) were included. As a control group, eight of the DCIS samples used in Figure 3B with a representative low degree of HEV remodeling were selected for the comparison with the triple‐negative breast cancer (TNBC) samples. The manual analysis was performed in random CCL21+ regions selected by hand. The same samples were run in HEV‐finder and the separate results were plotted to visualize the comparison.

Statistical analyses

Statistical analysis was performed using Graphpad Prism v. 7 (GraphPad Software, San Diego, CA, USA). The nonparametric Kruskal–Wallis test and Dunn's multiple comparisons test were used for analysis of more than two groups simultaneously.

Results

Validation of detection and classification of HEVs using HEV‐finder: a deep‐learning‐based tool

The outline and strategy for the design and operation of HEV‐finder is summarized in Figure 1. We trained the HEV‐finder on 12,524 HEVs manually classified as nondilated, intermediately dilated (lumen < 10 μm), or highly dilated (lumen > 10 μm) from our published dataset [3] (Figures 1 and 2A). HEV‐finder approaches detection and classification of HEVs in three steps (Figure 1), mimicking manual analysis [3]. An independent set of nine TDLNs (that had not been included for training) was used to validate the automatic HEV detection and classification by the HEV‐finder. HEV‐finder selected and classified random ROIs, and the same ROIs were classified manually to analyze the exact difference in the outcome of the analysis for each HEV, as illustrated in Figure 2B. The results of 1,791 identified HEVs from the nine LNs are presented in a confusion matrix (Figure 2C), showing that HEV‐finder recognizes highly dilated HEVs with high accuracy. Intermediately dilated HEVs have a dilation larger than 0 but smaller than 10 μm. The smallest lumens are inherently difficult to distinguish from closed (nondilated) vessels also for the human eye, and as expected these HEVs displayed the largest difference comparing the manual analysis to the HEV‐finder (Figure 2C, illustrations Figure 2B). This is also illustrated if the data from the confusion matrix (Figure 2C) is presented as the % total vessels (supplementary material, Table S4). Figure 3A (and supplementary material, Figure S1) show an example of the selection of ROIs within the CCL21+ paracortex, displayed in QuPath [14]. The HEV‐finder detects HEVs with high specificity (supplementary material, Table S5), small objects (aggregated antibodies), or signals on the borders of the image tiles were excluded from analysis.

Figure 1

Graphical abstract of the creation and workflow of HEV‐finder. The HEV‐finder was trained on 12,524 manually classified HEVs. The tool approaches detection and classification of HEVs in three steps. First, tiles are randomly placed within the paracortex (T‐cell zone) of the LNs, based on the marker CCL21. Next, HEVs are detected within the tiles and ROIs are created and classified as nondilated, intermediately dilated, or highly dilated. Finally, HEVs within 200 μm of adipocytes or metastasis are excluded. HEV locations, dilation status, and potential flags are saved to a spreadsheet. HEV‐finder also exports an image with tiles and outlines of HEVs, adipocyte regions, and tumor regions.

Computer‐aided automatic analysis is inherently less biased than the manual analysis. We therefore also compared automated and manual results at the patient level, not focusing on the exact same tiles, but accumulating classification results from different tiles in the samples. Figure 3B shows a dotplot of the proportion of highly dilated HEVs in TDLNs from patients with DCIS and IDC with and without metastasis, manually analyzed in our previous study [3] and here reanalyzed with HEV‐finder and the result plotted as the % highly dilated vessels. In contrast to the results we show in the confusion matrix (Figure 2C), where the same HEVs were analyzed manually and with the automatic HEV‐finder, it is not the same tiles and thereby not the same HEVs that are analyzed in Figure 3B. Selected patients with a high proportion of highly dilated HEVs are displayed in color for comparison (Figure 3B). These data demonstrate the robustness of the method in the ability to identify patients with a high and low degree of HEV remodeling and shows that the results are independent of the ROIs chosen for analysis.

In conclusion, our analysis validates that the HEV‐finder can identify and recognize dilated HEVs with high accuracy and can be used to identify different degrees of HEV remodeling in patients.

Application of the HEV‐finder on sections from patients with TN breast cancer

Our published analysis of HEVs in human breast cancer was focused on ER+ IDC compared to DCIS [3]. To evaluate the method in an independent patient cohort and a different subtype of breast cancer, we applied the HEV‐finder to a small cohort of patients with TN IDC, with and without LN metastasis. DCIS is a group of noninvasive breast cancer that is not classified for the expression of ER, progesterone receptor (PR), or human epidermal growth factor receptor 2 (HER2) in Sweden [15] and was therefore reused as a control group in this analysis. Similar to ER+ IDC, the TN breast cancer group displayed an increased proportion of highly dilated HEVs compared to DCIS and this effect was independent of metastasis (Figure 4A,B). The data confirm that manual analysis and AI‐based analysis by the HEV‐finder show consistent results on a group patient level (Figure 4B). These data further supports the robustness of HEV‐finder for evaluation of HEV dilation in TDLNs. Although larger follow‐up studies are warranted, the results support that TN and ER+ breast cancer share the ability to affect HEV functions within the draining axilla.

Figure 4

Application of the HEV‐finder on patients with triple negative (TN) breast cancer. (A) Representative immunofluorescence (IF) images of dilated HEVs (PNAd+/Claudin‐5+, magenta/cyan) in the paracortex (CCL21+, green) of nonmetastatic (left) and metastatic lymph nodes (LNs) draining a triple‐negative breast cancer (TNBC) primary tumor. Scale bar, 100 μm. (B) Analysis of the level of highly dilated HEVs in LNs draining ductal carcinoma in situ (DCIS) and TNBC, comparing manual versus automatic analysis by the HEV‐finder in samples not used for training. The four TNBC patients with the highest percentage of highly dilated HEVs are marked by separate symbols. Nonparametric Kruskal–Wallis test and Dunn's multiple comparisons test were used for statistical analysis. *p < 0.05, **p < 0.01.

Summary and discussion

New tools, including multispectral imaging, in situ RNA sequencing, and AI‐driven image analysis are defining a new era in human pathology [8, 9, 10]. We have here presented and validated the HEV‐finder, which is a tool for automated image analysis based on deep learning for the analysis of HEV dilation in TDLNs from cancer patients. Our approach to use a manually annotated dataset for training show a successful example of how deep learning can open up high‐throughput analysis of structural vascular and tissue morphology, changes that are not suitable to analyze with current commercial solutions, focused on single‐cell analysis or bright‐field microscopic tissue sections stained with hematoxylin and eosin.

Considering the cost and the risk of side effects of today's therapies for cancer, particularly in immunotherapy with low response rates [16, 17], it is of high clinical value to find additional treatment and diagnostic predictive biomarkers. It is already known that LN metastasis is one of the strongest predictive biomarkers for the risk of distant metastasis in several tumor types, including breast cancer [7].

Our application of the HEV‐finder on ER+ and TN breast cancer, together with data from the literature [1, 2], support that HEV dilation is a common cancer‐induced dysregulation of the LN functions in invasive cancer. We have previously demonstrated that these changes differentiate noninvasive DCIS from invasive IDC [3]. Notably, within the previous analysis of ER+ IDC [3] as well as within the TN breast cancer group we analyzed here (Figure 4), there is an interpatient heterogeneity. HEV‐dilation, alone or together with other biomarkers in the TDLNs, may prove to have value as a structural biomarker to further stratify patients with invasive cancer. The latter, and the full implications of HEV‐dysregulation for tumor immunity and for metastasis in different tumor types, still needs to be addressed. The HEV‐finder will facilitate these studies.

Vascular morphological changes in cancer are common not only in the TDLNs but also in the primary tumor [18], and we propose they could provide a new type of biomarker, particularly in combination with other immune oncology and stromal cell markers. A better biological understanding of vascular changes in cancer will require analysis over larger cohorts of patients, and thus solutions for automatic analysis are urgently needed. Our data emphasize that the development of these methods require carefully annotated data from manual analysis. HEV‐finder has the potential to be a useful tool for the future evaluation of HEV dilation as a biomarker in different cancers and to be a tool in studies aimed to understand mechanisms behind dysregulation of tumor immunity and promotion of cancer metastasis. To foster further development of similar methods, we provide HEV‐finder as an open‐source implementation and make all raw data available as training examples for teaching and development purposes.

Author contributions statement

MHU and TB conceived of the project. CA, AK and CW designed the HEV‐finder tool and performed the deep‐learning training. TB, SH and MFB did the manual image analysis and the validation of the HEV‐finder. DVB and FW selected the patient samples. MHU, TB and CA wrote the article. AK, CW and FW contributed to the review and editing of the article. All authors have read and agreed to the article.

Figure S1. Visualization of masks and annotations

Click here for additional data file.

Table S1. Clinical data of the patient cohort of triple negative breast cancer

Table S2. Primary antibodies

Table S3. Secondary antibodies

Table S4. Percent classified HEVs

Table S5. Sensitivity and specificity of HEV detection by the HEV‐finder

Click here for additional data file.