Artificial intelligence system of faster region-based convolutional neural network surpassing senior radiologists in evaluation of metastatic lymph nodes of rectal cancer.

BACKGROUND: An artificial intelligence system of Faster Region-based Convolutional Neural Network (Faster R-CNN) is newly developed for the diagnosis of metastatic lymph node (LN) in rectal cancer patients. The primary objective of this study was to comprehensively verify its accuracy in clinical use.
METHODS: Four hundred fourteen patients with rectal cancer discharged between January 2013 and March 2015 were collected from 6 clinical centers, and the magnetic resonance imaging data for pelvic metastatic LNs of each patient was identified by Faster R-CNN. Faster R-CNN based diagnoses were compared with radiologist based diagnoses and pathologist based diagnoses for methodological verification, using correlation analyses and consistency check. For clinical verification, the patients were retrospectively followed up by telephone for 36 months, with post-operative recurrence of rectal cancer as a clinical outcome; recurrence-free survivals of the patients were compared among different diagnostic groups, by methods of Kaplan-Meier and Cox hazards regression model.
RESULTS: Significant correlations were observed between any 2 factors among the numbers of metastatic LNs separately diagnosed by radiologists, Faster R-CNN and pathologists, as evidenced by rradiologist-Faster R-CNN of 0.912, rPathologist-radiologist of 0.134, and rPathologist-Faster R-CNN of 0.448 respectively. The value of kappa coefficient in N staging between Faster R-CNN and pathologists was 0.573, and this value between radiologists and pathologists was 0.473. The 3 groups of Faster R-CNN, radiologists and pathologists showed no significant differences in the recurrence-free survival time for stage N0 and N1 patients, but significant differences were found for stage N2 patients.
CONCLUSION: Faster R-CNN surpasses radiologists in the evaluation of pelvic metastatic LNs of rectal cancer, but is not on par with pathologists. TRIAL REGISTRATION: www.chictr.org.cn (No. ChiCTR-DDD-17013842).

Introduction

Rectal cancer is one of the most common gastrointestinal tumors. Although mortality from colorectal cancer has declined over the last 2 decades, it still stays high.[ Lymph node (LN) metastasis is the most common route by which rectal cancer spreads. Some studies have reported that LN metastasis increases the risk of loco-regional recurrence, and more than 40% of patients with LN metastasis suffer local recurrences without distant metastasis.[ Moreover, the risk of loco-regional recurrence, which has frequently been associated with a poor prognosis, is higher in patients with rectal cancer than those with colon cancer.[ According to the National Comprehensive Cancer Network (NCCN) Clinical Practice Guidelines in Oncology and some evidence, the patients’ treatment methods are directly determined by whether their pelvic LNs have metastasized, that is, whether they should be treated with radiotherapy and chemotherapy first then an operation or with excision first and then other therapies.[ Hence, an accurate evaluation of LN metastasis is crucial to clinical decision-making. Magnetic resonance imaging (MRI) is assumed to be an optimal diagnostic modality for tumor staging in rectal cancer patients due to its high soft-tissue contrast.[ Radiologists determine whether LN metastasis has occurred by observing the shape, boundary and signal intensity of the LNs via MRI.[ Nevertheless, accurate judgment in a short time is a great challenge for radiologists who have to integrate the aforementioned factors, especially when many patients are considered. In addition, the same MRI image may lead to very different conclusions when analyzed by different radiologists, and there is a relatively poor sensitivity for LN staging.[ As a consequence, it is often difficult to accurately determine whether LN metastasis has occurred.[ In recent years, the development of deep learning technology has greatly advanced image recognition capacity such that it is feasible to identify specific target areas within an image and allows the classification of images based on identified target features. This process is the same as in the diagnoses performed by radiologists, and a new solution to the aforementioned issues is provided. In January 2017, researchers at Stanford University successfully developed a deep learning algorithm, and its accurate recognition rate of skin cancer is on par with professional dermatologists.[ Although artificial intelligence (AI) system has been identified more accurate than senior physicians in the diagnosis of solid tumors, such as lung cancer, breast cancer and prostate cancer, few studies on metastatic LNs identification by AI have been reported yet.[ Compared with imaging-based diagnosis of solid masses, LN metastasis identification is more labor-intensive. As such, it is a clinically significant task to develop a reliable imaging recognition method for metastatic LN identification. Thus, we established a metastatic LN MRI database and an AI platform to perform repeated image training, whereby a deep learning model using Faster Region-based Convolutional Neural Network (Faster R-CNN) was developed. This model is designed as a tool for the rapid and accurate evaluation of LN metastasis. Our previous study showed details of the training and preliminary evaluation of the Faster R-CNN approach, and revealed that compared with the radiologist diagnoses, the area under the Faster R-CNN ROC was 0.912.[ Nevertheless, there are mild discrepancies of about 9% between Faster R-CNN and radiologists in imaging-based diagnoses. Moreover, no reports have clinically verified the accuracy of Faster R-CNN in the prognostic assessment of rectal cancer patients. In this study, 414 patients at various clinical centers were collected, and the accuracy of Faster R-CNN in the evaluation of metastatic LNs was methodologically and clinically verified in comparison with both radiologists and pathologists.

Methods

Ethical approval

The study was approved by the Ethics Committee and Institutional Review Board of Affiliated Hospital of Qingdao University, and all patients or their immediate families provided verbal informed consent which was recorded by telephone. The study was done in accordance with the International Conference on Harmonisation Guidelines for Good Clinical Practice and the Declaration of Helsinki.

Study design

In this study, 414 patients with rectal cancer discharged between January 2013 and March 2015 were collected from 6 clinical centers of China, including Affiliated Hospital of Qingdao University, the Sixth Affiliated Hospital of Sun Yat-Sen University, Fourth Hospital of Hebei Medical University, Qingdao Municipal Hospital, First Affiliated Hospital of Zhengzhou University and Beijing Friendship Hospital. The inclusion criteria were as follows: 1) subjects who had a definitive diagnosis of rectal cancer based on pathology; 2) subjects who received radical resection of rectal cancer; 3) subjects who had complete clinical information, pathological traits, and treatment plans, and the clinical and pathological attributes included age, sex, number of metastatic LNs diagnosed by senior radiologists, number of metastatic LNs diagnosed by pathologists, N staging based on radiologists, N staging based on pathologists, clinical staging based on radiologists, clinical staging based on pathologists, pathological type, tumor differentiation degree, status of intravascular tumor thrombus, and status of fascicular infiltration; and 4) subjects who received neither preoperative chemotherapies nor preoperative radiotherapies. The exclusion criteria were as follows: 1) subjects who were diagnosed with hereditary non-polyposis colorectal cancer (Lynch syndrome); 2) subjects who did not have complete medical records or records of treatment plans; 3) subjects who received palliative resection, with apparent intra-operative residual cancer tissue; and 4) subjects who had preoperative discoveries of other concurrent, primary malignant tumors (including those who had undergone surgery for other tumors).

Faster R-CNN evaluation

Faster R-CNN was evaluated after each iteration of training, and Faster R-CNN models after every 1000 iterations of training were used to recognize the MRI (GE Signa 3.0T HDX MR scanner and multi-channel phased array coil; the main scanning parameters and sequences are shown in Table 1) images in the test database. The results were compared with the unknown mark (truth value) of the metastatic lymph nodes to obtain the mAP of the Faster R-CNN models. The calculation method for mAP is shown in Eq. (1).

Table 1

Sequence parameters of magnetic resonance imaging (MRI) scans.

k is the number of images that have been identified, P(k) is the precision at the cut-off k in the list, and Δr(k) is the change in the recall from items k−1 to k. Q is the number of queries, where AveP is the average precision, specifically referring to the area under P-R curve. The calculation formula of AveP used the principle of calculus, taking the difference of recall ranging from k−1 to k as a small infinitesimal, which then multiplied the corresponding precision of k to obtain the area under the P-R curve. mAP value, the mean average precision, is the average AP value of multiple validation sets (for individual categories). As an indicator to measure the detection accuracy in object detection, each category can draw a P-R curve according to recall and precision. AP is the area under the curve, and mAP is the mean AP value of all categories. The MRI plain scan image for metastatic lymph nodes of each case was evaluated by Faster R-CNN, and the corresponding data were collected. Figure 1 shows the same metastatic LN separately recognized by radiologists (A), Faster R-CNN (B) and pathologists (C).

Figure 1

The same metastatic lymph node separately recognized by radiologists (A, T2 weighted imaging), Faster Region-based Convolutional Neural Network (Faster R-CNN) (B) and pathologists (C, Hematoxylin-Eosin staining, original magnification × 40).

Follow-up visits

All the patients were retrospectively followed up for recurrences of rectal cancer by telephone from January to March 2018. Records of their return visits were also checked via outpatient and inpatient electronic medical record systems, to verify information that the patients or their families supplied. The end point of follow-up visits was defined as recurrence of rectal cancer in 36 months after surgery; otherwise, it was considered a truncation. Fifty-two patients (12.6%) were lost during the follow-up, for neither the patients themselves or their families were successfully contacted by the telephone numbers registered in their medical records, nor their return visits was successfully tracked via electronic medical record systems. Finally, 362 cases were successfully followed up.

Registrations and code availability

The study was registered with www.chictr.org.cn, No. ChiCTR-DDD-17013842. Faster R-CNN Python code is available on Git-Hub at https://www.nature.com/authors/policies/data/data-availability-statements-data-citations.pdf.

Statistical analysis

Qualitative baseline characteristics were expressed as frequency (proportion); quantitative baseline attributes were expressed as mean ± standard deviation (SD). Bilateral t-test or χ test was used to compare the differences between the recurrence and recurrence-free groups. In methodological verifications, correlations between any 2 factors among the numbers of metastatic lymph nodes separately diagnosed by radiologists, Faster R-CNN and pathologists were examined by method of Pearson correlation; N staging separately based on radiologists, Faster R-CNN and pathologists were mutually compared by method of consistency check. In clinical verifications, methods of Kaplan-Meier and Cox hazards regression model were used to explore factors that may affect the recurrence-free survivals of rectal cancer patients. The patients were divided into 3 groups based on the corresponding diagnostic methods, namely the radiologist, Faster R-CNN and pathologist groups, and the recurrence-free survivals of patients with the same N stage were compared among the 3 groups by method of Kaplan-Meier. The hazard ratio (HR) and 95% confidence interval (CI) were used to express the association between a factor and the clinical outcome. All statistical analyses were performed using SPSS version 16.0 (SPSS Inc., Chicago, United States). A P < 0.05 indicated statistical significance.

Results

Baseline characteristics of the follow-up patients

A total of 362 patients were successfully followed up, including 226 males (62.4%) and 136 females (37.6%), with an age range from 26 to 85 years (mean age 58.0 ± 12.4 years). All the patients were diagnosed with rectal adenocarcinoma, and 12.2% of them suffered recurrences. Significant differences were found in the distributions of sex, tumor differentiation degree, N staging, clinical staging, and operation methods between the recurrence and recurrence-free groups (P < 0.05; Table 2)

Table 2

Baseline characteristics between the recurrence group and recurrence-free group of the follow-up patients.

Correlation analyses of metastatic LNs separately diagnosed by Faster R-CNN, radiologists and pathologists

Pair-wise correlation analyses were separately performed between any 2 factors among the numbers of metastatic LNs diagnosed by radiologists, number of metastatic LNs diagnosed by pathologists, and number of metastatic LNs diagnosed by Faster R-CNN. The results showed that there were significant correlations between any 2 of the 3 factors, as evidenced by rradiologist-Faster R-CNN of 0.912 (P < 0.001), rPathologist-radiologist of 0.134 (P = 0.011), and rPathologist-Faster R-CNN of 0.448 (P < 0.001).

Consistency check of N staging separately based on Faster R-CNN, radiologists and pathologists

Faster R-CNN was highly consistent with radiologists in N staging of the patients, and the value of kappa coefficient was 0.926 (P = 0.018). The kappa value between Faster R-CNN and pathologists was 0.573 (P = 0.039), and this value between radiologists and pathologists was 0.473 (P = 0.043; Table 3).

Table 3

Pair-wise consistency check of N staging separately based on Faster R-CNN, radiologists and pathologists.

Verification of faster R-CNN in the prognostic assessment of patients with rectal cancer

Univariate survival analyses

The Kaplan-Meier method was used to compare the survival curves of patients of different sexes, N stages based on radiologists, N stages based on Faster R-CNN, N stages based on pathologists, clinical stages based on radiologists, clinical stages based on pathologists, degree of tumor differentiation, and operation methods. It showed that sexes (χ = 4.509, P = 0.032), N staging based on pathologists (χ = 28.994, P < 0.001, Figure 2C), clinical staging based on pathologists (χ = 13.041, P = 0.005), degree of tumor differentiation (χ = 37.338, P < 0.001), and operation methods (χ = 13.122, P = 0.000) significantly affected the postoperative recurrence-free survival time (RFST), but N staging based on radiologists (χ = 5.643, P = 0.060, Figure 2A) and N staging based on Faster R-CNN (χ = 5.828, P = 0.054, Figure 2B) were not significantly associated with the RFST of rectal cancer patients.

Figure 2

Recurrence-free survival curves of the patients with rectal cancer of different N stages separately based on (A) radiologists; (B) Faster R-CNN; (C) pathologists.

Multivariate survival analyses

The association between RFST and variables of sex, N staging based on radiologists, N staging based on Faster R-CNN, N staging based on pathological diagnosis, clinical staging based on pathologists, degree of tumor differentiation, and operation methods were analyzed together by Cox hazards regression model for multivariable analyses.

Multivariate survival analysis not stratified by sexes

The results showed that sex, operation methods, degree of tumor differentiation, and N staging based on pathologists significantly affected the RFST of patients, as evidenced by the corresponding HR and 95% CI values of 2.340 (1.179, 4.867), 3.552 (1.717, 7.349), 2.552 (1.518, 4.289), and 2.607 (1.374, 4.944) respectively [Table 4].

Table 4

Baseline characteristics associated with the postoperative recurrences of rectal cancer patients analyzed by multivariate survival analyses.

Multivariate survival analysis stratified by sexes

Considering the sex differences of rectal cancer incidence, multivariate analysis stratified by sexes was performed. The results showed that operation methods, degree of tumor differentiation, and N staging based on pathologists significantly affected the RFST of patients, as evidenced by the corresponding HR and 95% CI values of 3.521 (1.704, 7.275), 2.591 (1.539, 4.363), and 2.601 (1.374, 4.926) respectively [Table 5].

Table 5

Baseline characteristics associated with the postoperative recurrences of rectal cancer patients analyzed by multivariate survival analyses stratified by sexes.

Comparison of the survivals of patients at the same N stage separately assessed by radiologists, Faster R-CNN and pathologists

The patients were divided into 3 groups based on the corresponding diagnostic methods, namely the radiologist, Faster R-CNN, and pathologist groups. The recurrence-free survivals of patients at the same N stage were compared among the 3 groups. The results showed that the 3 groups displayed no significant differences in the recurrence-free survivals for patients assessed as stage N0 and stage N1 (stage N0: χ = 0.014, P = 0.993, the 3 cures were almost completely overlapped; stage N1: χ = 2.314, P = 0.314, Figure 3A). However, for stage N2 patients, as shown in Figure 3B, the 3 groups showed significant differences in the recurrence-free survivals (χ = 9.344, P = 0.009); pair-wise comparisons for each pair of the 3 groups revealed that significant differences were observed between the radiologist and pathologist groups (χ = 7.653, P = 0.006) as well as between the Faster R-CNN and pathologist groups (χ = 6.190, P = 0.013), whereas the Faster R-CNN and radiologist groups exhibited no significant difference in the recurrence-free survivals (χ = 0.206, P = 0.650).

Figure 3

Recurrence-free survival curves of the patients with rectal cancer at (A) stage N1 separately assessed by radiologists, Faster R-CNN and pathologists; (B) stage N2 separately assessed by radiologists, Faster R-CNN and pathologists.

Discussion

The focus of this study was to comprehensively verify the accuracy of Faster R-CNN AI system in the evaluation of pelvic metastatic LNs of rectal cancer. Faster R-CNN, proposed by Ross Girshick in 2016, is a novel target detection algorithm based on R-CNN and Fast R-CNN.[ In comparison with other deep learning algorithms, Faster R-CNN introduces the concept of region proposal network, which is built on top of additional convolutional features. This network shares the convolution feature of an entire image with region-based detection network. Faster R-CNN generates a region proposal network in Fast R-CNN, which effectively streamlines a large volume of repetitive computations and correspondingly facilitates fast, real-time target recognition. As such, Faster R-CNN has become the most favorable technology in the field of AI-based automatic recognition.[

As known, neo-adjuvant chemotherapy or radiotherapy leads at least 20% of rectal cancer patients to pathological complete regression, and 40% of patients experience TN down-staging, therefore patients who received preoperative chemotherapy or preoperative radiotherapy were excluded from the study. In this study, the recurrence and recurrence-free groups had very different baseline characteristics. As shown in Table 2, sex, tumor differentiation degree, N staging, clinical staging, and operation methods were significantly associated with the recurrence status of rectal cancer patients. Of all the risk factors mentioned, N staging was the most significant. Patients at stage N2 were about 4.3-fold more likely than those at stage N1, and about 18.4-fold more likely than those at stage N0 to suffer tumor recurrences in 36 months after radical surgery, therefore accurate diagnosis of metastatic LNs in rectal cancer patients is on top priority.

Our previous studies showed that Faster R-CNN was highly consistent with radiologists in the diagnosis of metastatic LNs, but there still were mild discrepancies of about 9%.[ Moreover, it is more difficult to retrospectively mark pathologist diagnosed metastatic LNs than solid tumors in MRI data, for the large quantity of LNs and tiny differences among LNs, such that the training of Faster R-CNN on recognition of LNs in MRI data is radiologist-based, but not pathologist-based. Thus, pathologist diagnoses of metastatic LNs and the survival index of RFST were further introduced for comprehensive assessment on the accuracy of Faster R-CNN in the evaluation of metastatic LNs. Correlation analyses showed that the numbers of metastatic LNs separately diagnosed by radiologists, Faster R-CNN and pathologists exhibited pair-wise correlations. Of note, the rradiologists-Faster R-CNN value of 0.912 is highly consistent with our previous finding; when compared with the pathologist diagnoses, the rPathologist-Faster R-CNN value of 0.448 is greater than the rPathologist-radiologist value of 0.134. It indicated that Faster R-CNN diagnoses were more accordant with pathologists’. Moreover, for N staging check, the kappa coefficient between Faster R-CNN and pathologists was 0.573, bigger than 0.473 between radiologists and pathologists. 72 patients (19.9%) with rectal cancer of stage N1 were misclassified into stage N2 by Faster R-CNN, with 10 (2.8%) of stage N2 misclassified into stage N1, 4 (1.1%) of stage N1 into stage N0, and 2 (0.5%) of stage N2 into stage N0. It suggested Faster R-CNN tended to increase the magnitude of N staging in rectal cancer patients. Faster R-CNN was highly accordant with radiologists in N staging assessment, as evidenced by the kappa coefficient of 0.926. The mild discrepancies between Faster R-CNN and radiologists were attributed to the assessment on patients at stage N2. As shown in Table 3, 42 patients were correctly classified into stage N2 by Faster R-CNN, whereas only 26 patients were correctly classified into stage N2 by radiologists, and 16 patients at stage N2 were misclassified into stage N1 by radiologists compared with Faster R-CNN.

For clinical verifications, recurrence-free survival analyses were done by methods of Kaplan-Meier and Cox hazard regression model. Univariate survival analyses showed that sex, N staging based on pathologists, clinical staging based on pathologists, degree of tumor differentiation, and operation methods significantly affected the postoperative RFST, but N staging based on radiologists and N staging based on Faster R-CNN did not reach statistical levels. It indicated that pathological examination was still the most reliable method to diagnose metastatic LNs and to evaluate the prognosis of rectal cancer patients, but unfortunately, it was a postoperative method; Radiologists and Faster R-CNN owned their advantages of preoperative diagnoses. As shown in Figure 2, all the 3 survival curves demonstrated the same tendency that patients at higher N stage had lower recurrence-free survival rates. The most differences between the survival curves separately based on Faster R-CNN and pathologists, as well as between those separately based on radiologists and pathologists, were that patients at stage N2 assessed by pathologists had significantly lower accumulative recurrence-free survival rate of 65%, compared with those separately assessed by Faster R-CNN and radiologists who both had a accumulative recurrence-free survival rate of 85%. Multivariate survival analyses either stratified or not stratified by sexes showed that degree of tumor differentiation and operation methods influenced the recurrence-free survivals of patients. With every degree increment in tumor malignancy, the risk of recurrence increased by 2.6-fold. Patients who received radical surgery of Miles were 3.5-fold more likely than those receiving surgery Dixon to suffer recurrences of rectal cancer. N staging based on pathologists was independently associated with the recurrence-free survivals of patients; with every degree increment in N staging, the risk of recurrence increased by 2.6-fold. In addition, male patients took higher risks of suffering tumor recurrences than the females, and the HR value was 2.396.

According to the 3 evaluation methods used, the patients were divided into radiologist, Faster R-CNN, and pathologist groups. The recurrence-free survivals of patients at the same N stage were compared among the 3 groups. The results revealed that the 3 groups showed no significant survival differences in patients at either stage N0 or stage N1, but the differences were significant in patients at stage N2. The RFST of patients at stage N2 was much shorter in the pathologist group than the Faster R-CNN group and the radiologist group, but no significant differences were found between the Faster R-CNN group and the radiologist group. It suggested that compared with pathologists, both of Faster R-CNN and radiologists provided relatively comparable evaluation on the prognosis of rectal cancer patients, but tended to increase the magnitude of N staging and to predict a worse prognosis. This condition probably contributes to the rising attention of doctors, but may also affect the treatment schedules for patients. Doctors when using Faster R-CNN for metastatic LNs diagnosis and preoperative N staging should take this condition into account. Moreover, our previous studies showed that the diagnosis time of Faster R-CNN was 20 s per case, which is much shorter than the average time (600 s per case) of radiologists. In summary, we present the methodological and clinical verifications of Faster R-CNN on the evaluation of pelvic metastatic LNs in patients with rectal cancer. Faster R-CNN has preoperative advantages for LN diagnosis; it is more accurate and efficient than radiologists to diagnose metastatic LNs and to evaluate the prognosis of patients, but not on par with pathologists.

Acknowledgements

We are deeply grateful to all of the participants and to the doctors who contributed to this project.

Funding

This work was supported by the grants from the Twelfth Five-Year Plan for Science and Technology Support of China (No. 2013BFASTER R-CNN01B03), the Major Science and Technology Project of Independent Innovation of Qingdao, China (No. 14-6-1-6-zdzx), and the Key Research and Development Plan (tackle hard-nut problems in science and technology) of Shandong Province (No. 2018GSF118206).

Conflicts of interest

None.