Computer-aided diagnosis system of thyroid nodules ultrasonography: Diagnostic performance difference between computer-aided diagnosis and 111 radiologists.

To evaluate the diagnostic efficiency of computer-aided diagnosis (CAD) system and 111 radiologists with different experience in identifying benign and malignant thyroid nodules, and to summarize the ultrasound features that may affect the diagnostic of CAD and radiologists.Fifty thyroid nodules and 111 radiologists were enrolled in this study. All the 50 nodules were diagnosed by the 111 radiologists and the CAD system simultaneously. The diagnostic performance of the CAD system, senior and junior radiologists with the maximum accuracy were calculated and compared. Interobserver agreement for different ultrasound characteristics between the CAD and senior radiologist were analyzed.CAD system showed a higher specificity than junior radiologist (87.5% vs 70.4%, P = .03), and a lower sensitivity than the senior radiologist and junior radiologist but the statistics were not significant (76.9% vs 86.9%, P > .5; 76.9% vs 82.6%, P > .5). The CAD system and senior radiologist got larger AUC than junior radiologist but the differences were not statistically significant (0.82 vs 0.76, respectively; P = .5). The interobserver agreement for the US characteristics between the CAD system and senior radiologist were: substantial agreement for hypoechoic and taller than wide (kappa value = 0.66, 0.78), and moderate agreement for irregular margin and micro-calcifications (kappa value = 0.52, 0.42).The CAD system achieved equal diagnostic accuracy to the senior radiologists and higher accuracy than the junior radiologists. The interobserver agreements in the US features between the CAD system and senior radiologist were substantial agreement for hypoechoic and taller than wide; moderate agreement for irregular margin and micro-calcifications. The location of a thyroid nodule and the feature of macrocalcification with wide acoustic shadow may influence the analysis of the CAD system.

Introduction

According to the newest comprehensive cancer statistics, the incidence of thyroid cancer has been rising in China, especially among women with an incidence of 67.9, but thyroid cancer has a lower mortality rate than other cancers.[ Epidemiologic studies have shown the prevalence of palpable thyroid nodules to be approximately 5% in women and 1% in men living in iodine-sufficient parts of the world.[ In clinical practice, thyroid nodules are classified into malignant and benign. Surgical is not necessary for benign lesion unless the nodule cause some difficulties such as pain and bump in the anterior region of the neck, and dysphagia due to the enlarging of the nodule. Papillary thyroid cancer and follicular cancer accounts for mostly in malignant nodules, surgical treatment is required in malignant thyroid nodule. Thus, differentiated thyroid malignant and benign nodules have a great influence on thyroid therapy.

Examination of thyroid lesions includes neck palpation, ultrasonography (US), computed tomography (CT), and magnetic resonance imaging (MRI). US-guided fine-needle aspiration biopsy (FNA) is the golden standard to distinguishing the malignant and benign thyroids. Among them, with the rapid development of ultrasound technology, high-frequency US plays the most important role in distinguishing malignant and benign thyroid nodules by providing high-resolution real-time images with less expensive and radiation-free than other imaging methods. This is important for the diagnosis and treatment of patients with thyroid nodules. The main high-risk sonographic characteristics of malignant nodules on ultrasound include hypoechoic, irregular shape, speculated margin, microcalcifications, taller-than-wide orientation. The sensitivity and specificity of ultrasound screening for malignant thyroid glands range from 48.0% to 94.3% and from 53.0% to 93.0%.[ However, the interpretation of the ultrasound images highly depends on radiologist experience. Therefore, ultrasound has not been specific enough to be an independent objective indicator for screening for malignant thyroid nodules.

In many areas, such as the detection of pulmonary nodules and thyroid nodules, computer-aided diagnostic (CAD) systems have been developed to obtain accurate, repeatable, and more objective diagnostic results. Many previous studies have been demonstrated that CAD can improve the detection of pulmonary nodules in the context of 2 chest radiography,[ CT[ and even on chest tomosynthesis images.[ Especially among interns and residents,[ the diagnostic performance using CAD is significantly better than that without CAD. The CAD diagnostic system for thyroid nodules is based on different features such as sonographic features, texture, tissue stiffness score, vascular indices, and even contrast-enhanced ultrasonography. According to previous studies,[ the accuracy of the CAD reached 81.5% to 100%, and the sensitivity was from 72.4% to 100%. The purpose of this study was to evaluate the diagnostic performance in CAD system, senior and junior radiologists, and to assess the sensitivity of different sonographic features in radiologists and CAD system.

Methods

This study was approved by the Institutional Review Board and Ethics Committee of Sichuan Cancer Hospital. Obtaining informed consent is waived or exempted for this study.

Thyroid nodules ultrasound images

Ultrasound images of thyroid nodules with pathology confirmed by FNA results or postoperative pathology were selected from our picture archiving and communication system (PACS) from February 2016 to June 2018, and these images were performed by many radiologists scanned. Inclusion criteria for ultrasonic images: linear probes of different brands of ultrasonic instruments were used to collect two-dimensional ultrasonic images of thyroid nodules; thyroid nodules were clearly displayed in ultrasound images; contains the entire nodule; normal thyroid tissue were existed around the nodule; no superposition of blood flow signal in ultrasonic image; nodules were pathologically certificated. Exclusion criteria for ultrasonic images: unclear thyroid nodules in ultrasonic images; the nodules were displayed incompletely; no normal thyroid tissue around the nodule; nodules were not certificated by pathology.

Radiologists and CAD system

111 radiologists from different hospitals were invited to participate in an academic conference on the diagnosis of thyroid nodules. We conducted a study to compare senior and junior Radiologists with the CAD system (AmCAD-UT, AmCad BioMed Corporation, Taipei City, Taiwan), this system has been assessed and certified as a medical device and has been approved for human diagnosis, with image analysis technology used to diagnose the features of thyroid lesions on the basis of 2D US images). They read the 50 US images and tell the malignant or benign simultaneously and provide an overall diagnosis for the nodules, and we calculated the accuracy of each radiologist and CAD system.

The diagnostic efficacy discrepancies in junior radiologists (with US thyroid nodule diagnosis experience <30,000 cases), senior radiologists (with US thyroid nodule diagnosis experience >30,000 cases) with maximum accuracy and CAD system were compared. The pathology result of each nodule was both blind to radiologists and CAD system.

All the CAD diagnoses were performed by the trained engineer from the CAD system company. AmCAD-UT is a software system that is designed to facilitate the detection, visualization, and characterization of thyroid nodule features on sonographic images by use of statistical pattern recognition and quantification algorithms. The procedures of CAD system were listed as follows: import the ultrasound images into the CAD system, draw the ROI of the nodules, and then click “Analyze” button.[ The CAD system evaluates nodule echogenicity (hyper/iso/hypo-echogenicity), echogenic foci, margin (irregular, ill-defined, halo), taller than wide, texture (homogeneous, heterogeneous), and then, analyzes the risk of malignancy on the basis of the quantified results obtained.

Statistical analysis

Data were analyzed using SPSS 22.0 (SPSS Inc., Chicago, IL). The measurement data were expressed as mean ± standard deviation (mean ± SD), and the counting data were expressed as cases. By testing against pathological results, the diagnostic sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and accuracy of radiologists with maximum accuracy and CAD for differentiation between benign and malignant thyroid nodules were calculated based on every lesion. Receiver operating characteristic (ROC) curve and AUC value were used to examine the diagnostic effect of the senior radiologist with maximum accuracy, the junior radiologist with maximum accuracy, and CAD system. Interobserver agreements for different ultrasound characteristics between CAD system and the senior radiologist were assessed with Cohen kappa statistic. The values were interpreted as the following way: poor (<0.20), fair agreement (0.21–0.40), moderate (0.41–0.60), substantial (0.61–0.80), and almost perfect agreement (0.81–1.0).

Results

A total of 50 US images were recruited in this study, the pathological results of these lesions included: 23 malignant thyroid nodules (22 papillary carcinomas, and 1 medullary carcinoma), 27 benign lesions (17 adenomas, 3 nodular goiters, and 7 inflammatory lesions [Table 1]).

Table 1

Radiologists and nodules characteristics.

Among the 111 radiologists, 50 were senior and 61 were junior. The diagnostic accuracy of each radiologist was calculated and set accuracy for 60% as the reference standard. Fifty one radiologists with accuracy >60% and were constituted by 39 senior radiologists and 12 junior radiologists. The rest of 60 radiologists with accuracy <60% covered 11 senior radiologists and 49 junior radiologists (Table 1).

The sensitivity, specificity, PPV, NPV, and overall accuracy for CAD, the senior radiologist with maximum accuracy and junior radiologist with maximum accuracy are reported in Table 2. The CAD system achieved sensitivity, specificity, PPV, NPV, and overall accuracy of 76.9%, 87.5%, 86.9%, 77.8%, and 82.0%. The senior radiologist with maximum accuracy achieved sensitivity, specificity, PPV, NPV, and overall accuracy of 86.9%, 77.8%, 76.9%, 87.5%, and 82.0%. The junior radiologist with maximum accuracy achieved sensitivity, specificity, PPV, NPV, and overall accuracy of 82.6%, 70.4%, 70.4%, 82.6%, and 76%. In ROC curve analysis, the area under the curve (AUC) was 0.82, 0.82, and 0.76 for CAD, Senior, and Junior separately (Fig. 1). The CAD system showed a higher specificity than junior radiologist (87.5% vs 70.4, P = .03), while a lower sensitivity than the senior radiologist and junior radiologist but the statistics were not significant (76.9% vs 86.9%, P > .5; 76.9% vs 82.6%, P > .5); CAD system and senior radiologist got larger AUC in differential diagnosis of thyroid nodules than junior radiologist, however, the differences were not statistically significant (0.82 vs 0.76, respectively; P = .5)

Table 2

The diagnostic performance of the CAD system, senior radiologist with maximum accuracy, and junior radiologist with maximum accuracy for thyroid nodules diagnostic.

Figure 1

Receiver operating characteristic (ROC) curves. Receiver operating characteristic curves for CAD system (AUC, 0.82), senior radiologist with maximum accuracy (AUC, 0.82), and junior radiologist with maximum accuracy (AUC, 0.76). AUC = area under the curve; CAD = computer-aided diagnosis.

Furthermore, we analyzed the accuracy of each US image in radiologists and set 60% as the reference standard. In terms of diagnostic accuracy >60%, there were 27 thyroid nodules with accuracy >60%, 15 were malignant and 12 were benign nodules, we found that “hypoechoic” (15 in 15), “microcalcifications” (8 in 15), “taller than wide” (7 in 15), and “irregular margin” (11 in 15) were the US features with the highest accuracy to detect malignant thyroid nodules. The US features “isoechoic” (10 in 12) and “spongiform” (6 in 12) were the main characteristics to indicate the benign thyroid nodules. The rest of 23 nodules with diagnostic accuracy <60% concluded 14 benign nodules and 9 malignant nodules. For radiologists, the nodules in Hashimoto's thyroiditis were harder to distinguish regardless of malignant or benign ones (9 in 23); small hypoechoic nodules (<5 mm) with colloid inside the lesion might be misinterpreted as benign ones (3 in 9) (Fig. 2).

Figure 2

The misdiagnosed thyroid nodules of radiologists. A. A small papillary carcinoma with colloid inside the lesion (arrow) was misdiagnosed as benign one by most radiologists (with an accuracy of 20.9%); B. A papillary carcinoma nodule (arrow) in Hashimoto thyroiditis was misdiagnosed as benign one by most radiologists (with an accuracy of 24.0%).

For CAD system, a total of 7 thyroid nodules were misdiagnosed. The location of atypical nodules might influence the diagnostic accuracy such as when they are located in the isthmus, near the thyroid capsule or adjacent to the trachea (5 in 7) (Fig. 3); Macrocalcification inside the lesion with wide acoustic shadow may also influence the analysis of the CAD system (2 in 7).

Figure 3

The misdiagnosed thyroid nodules of CAD system. A nodular goiter located near to the thyroid capsule and trachea was misdiagnosed as malignant nodule by CAD system. CAD system indicated that this nodule is markedly hypoechoic and heterogeneous nodule with the presence of echogenic foci and taller than wide shape. CAD = computer-aided diagnosis.

Interobserver agreement in the US features with the highest accuracy to detect malignant thyroid nodules between CAD system and senior radiologist with maximum accuracy were analyzed as followed: substantial agreement for hypoechoic (kappa value = 0.66) and taller than wide (kappa value = 0.78); moderate agreement was seen for irregular margin (kappa value = 0.52) and micro-calcifications (kappa value = 0.42); heterogeneous was the only character in fair agreement (kappa value = 0.33) (Table 3).

Table 3

Interobserver agreements for different ultrasound features between CAD system and senior radiologist.

Discussion

Diagnostic thyroid US should be performed in all patients with a suspected thyroid nodule according to the newest American Thyroid Association (ATA) Management Guidelines.[ Although thyroid ultrasound has been widely used to stratify the risk of malignancy in thyroid nodules, it is not easy for junior radiologists to interpret the US images accurately due to lacking abundant experience compared with senior radiologists. CAD system devices are designed to differentiate nodules automatically, which have been used mainly in lung nodules, thyroid nodules, breast tumors diagnosis, and achieved good diagnostic results.[ CAD system might increase the accuracy for the junior radiologist. However, a study related to the comparison of accuracy between CAD and radiologists with different clinical experience is still rare.

Our study showed that CAD system achieve the equal diagnostic accuracy compared with senior radiologist, and a higher specificity than junior radiologist (87.5% vs 70.4%, P = .03). CAD system showed a lower sensitivity than the senior radiologist and junior radiologist but the statistics were not significant (76.9% vs 86.9%, P > .5; 76.9% vs 82.6%, P > .5). These results are consistent with previous studies by using the CAD system to differentiate malignant and benign thyroid nodules (from 80% to 94.8% in sensitivity and from 85% to 91% of accuracy).[ CAD system and senior radiologist got larger AUC in the differential diagnosis of thyroid nodules than junior radiologist, although the differences were not statistically significant (0.82 vs 0.76, respectively; P = .5), this result was in accordance with Jeong results in general.[ Therefore, we make a hypothesis that the CAD system may provide aided value for junior radiologist with higher specificity and positive predictive value.

In addition, we calculated the accuracy of each nodule among all radiologists, with a majority of senior radiologists with an accuracy of >60% (76.5% are senior radiologists and the remaining 23.5% are junior radiologists). In contrast, in nodules, junior radiologists account for the majority with <60% accuracy (18.3% for senior radiologists and 81.7% for junior radiologists). Therefore, CAD system showed better performance than junior radiologists in diagnosing thyroid nodules, and our results again emphasize the importance and necessity of accumulating experience and learning curves.

In the analysis of accuracy in each nodule, our study showed that “hypoechoic,” “microcalcifications,” “taller than wide,” and “irregular margin” were the main US features with the highest accuracy to detect malignant thyroid nodules for radiologists. The US features “isoechoic” and “spongiform” indicate benign thyroid nodules mostly for radiologists. These results were consistent with previous studies talking about the diagnostic accuracy of different thyroid US features in detecting thyroid cancer.[ The interobserver variability results in the US features between the CAD system and senior radiologists were substantial agreement for hypoechoic and taller than wide; moderate agreement was seen for irregular margin and micro-calcifications; fair agreement for heterogeneous. These results were different from Young conclusion which studied another commercial CAD system and showed the margin definition showed a fair agreement (kappa = 0.239) and substantial agreement for composition, orientation, echogenicity (kappa = 0.659, 0.740, 0.733, respectively).[

In the nodules with accuracy <60% for all radiologists, we found that the nodule in Hashimoto thyroiditis was difficult to distinguish regardless of malignant or benign one. Small malignant hypoechoic nodules (<5 mm) with colloid inside the lesion might be misinterpreted as benign nodules. However, the CAD system can provide an accurate diagnostic without being affected by the background of Hashimoto thyroiditis.

For CAD system, according to our results and further analysis, the location of atypical nodule might affect the diagnostic. Most of the misdiagnosed nodules by CAD system were located in the isthmus or near the thyroid capsule. Furthermore, macrocalcification inside the lesion with wide acoustic shadow may also affect the analysis of the CAD system. However, radiologists can identify the macrocalcification without being affected by the location of the nodules.

There were some limitations in our study. Firstly, a total of 50 thyroid nodule images were not enough. Future studies should include a larger sample size of nodules. Second, we didn’t discuss the diagnostic improvement level of junior radiologist with the aid of the CAD system, so these data should be further evaluated.

In conclusion, the CAD system achieved similarly diagnostic accuracy compared with the senior radiologist, and the CAD system showed a higher specificity than the junior radiologist but a lower sensitivity than senior radiologist and junior radiologist. The interobserver agreement in the US features between the CAD system and senior radiologist were the substantial agreement for hypoechoic and taller than wide; moderate agreements were seen for irregular margin and micro-calcifications. The location of the thyroid nodule and macrocalcification inside the lesion with wide acoustic shadow may affect the analysis of the CAD system.

Author contributions

Conceptualization: Tingting Li, Man Lu.

Formal analysis: Ting Wei, Xueqing Cheng.

Methodology: Tingting Li, Juan Li, Shibin Zou.

Software: Minggang Wu, Lu Wang, Ziyue Hu, Zirui Jiang.

Writing – original draft: Tingting Li.

Writing – review & editing: Tingting Li, Man Lu.