A systematic review and meta-analysis of diagnostic performance and physicians' perceptions of artificial intelligence (AI)-assisted CT diagnostic technology for the classification of pulmonary nodules.

BACKGROUND: Lung cancer was the second most commonly diagnosed cancer and the leading cause of cancer death in 2020. Although artificial intelligence (AI)-assisted diagnostic technologies have shown promise and has been used in clinical practice in recent years, no products related to AI-assisted CT diagnostic technologies for the classification of pulmonary nodules have been approved by the National Medical Products Administration in China. The objective of this article was to systematically review the diagnostic performance of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant and to analyze physicians' perceptions of this technology in China.
METHODS: All relevant studies from 6 literature databases were searched and screened according to the inclusion and exclusion criteria. Data were extracted and the study quality was assessed by two reviewers. The study heterogeneity and publication bias were estimated. A questionnaire survey on the perceptions of physicians was conducted in 9 public tertiary hospitals in China. A meta-analysis, meta-regression and univariate logistic model were used in the systematic review and to explore the association of physicians' perceptions with their rate of support for the clinical application of the technology.
RESULTS: Twenty-seven studies with 5,727 pulmonary nodules were finally included in the meta-analysis. We found that the quality of the included studies was generally acceptable and that the pooled sensitivity and specificity of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant were 0.90 and 0.89, respectively. The pooled diagnostic odds ratio (DOR) was 70.33. The majority of the surveyed physicians in China perceived "reduced workload for radiologists" and "improved diagnostic efficiency" as the important benefits of this technology. In addition, diagnostic accuracy (including misdiagnosis) and practical experience were significantly associated with whether physicians supported its clinical application.
CONCLUSIONS: In the context of lung cancer diagnosis, AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant has good diagnostic performance, but its specificity needs to be improved. 2021 Journal of Thoracic Disease. All rights reserved.

Introduction

Lung cancer was the second most commonly diagnosed cancer and the leading cause of cancer death in 2020, representing approximately one in 10 (11.4%) cancers diagnosed and one in 5 (18.0%) cancer deaths and accounting for an estimated 2.2 million new cancer cases and 1.8 million deaths (1). The five-year survival rate for lung cancer ranges from 70% for stage I to less than 5% for stage IV (2). A number of surgical and medical therapies can cure many cases of small localized tumors, but only 15% of patients with lung cancer are diagnosed at the localized stage (3). In addition, lung cancer treatment is costly, with approximately $9.6 billion being spent on this treatment in the United States each year (4).

Low-dose computed tomography (CT) has been widely used for lung cancer screening (5). It now offers the ability to detect small lesions (less than 10 mm in diameter) and has the potential to detect stage I tumors 4 to 6 times as frequently as conventional radiography (6). Many current guidelines for lung cancer screening recommend annual low-dose chest CT screening for high-risk individuals, for whom the benefit of low-dose chest CT screening outweighs its harms (7-9), and recommend lung cancer diagnosis is based on the size and attenuation characteristics of the nodule as well as the presence of lung cancer risk factors for small nodules and on the estimated probability of malignancy and the yield of additional testing for larger nodules (10,11).

However, with CT being used increasingly more often in healthy people and patients with suspected lung diseases, radiologists face a greatly increased workload regarding the assessment of pulmonary nodules [defined as a focal opacity of <3 cm in diameter (12)] in CT images. Moreover, one study showed that the sensitivity of radiologists in accurately identifying and classifying pulmonary nodules (as benign or malignant nodules) ranged from 30% to 97%, and the false positive rate was as high as 2.1 per scan (13). The differentiation of benign and malignant nodules is a challenging task and requires a combination of visual assessments and measurements. Different physicians may also have different interpretations.

Artificial intelligence (AI) holds great promise and has been used in clinical practice in recent years (14,15). One example is AI-assisted diagnostic technologies, which use computerized extraction and classification algorithms to identify and classify diseases. In general, AI-assisted diagnostic technology has four functions: preprocessing images (cleaning images and removing noise); segmenting the region of interest (ROI); extracting and selecting the most discriminative features; and classifying the disease according to the features (16). Machine learning algorithms (MLAs), such as artificial neural networks (ANNs) (17-20), support vector machines (SVMs) (21-24), deep belief networks (DBNs) (25-27), Bayesian networks (BNs) (28-30), convolutional neural networks (CNNs) (31-33), and decision trees (DTs) (34-37), are used in AI-assisted CT diagnostic technologies to detect and classify pulmonary nodules as benign or malignant.

AI-assisted diagnostic technology is being developed, explored and evaluated to support clinical diagnosis and treatment. It is used as a “second opinion” to assist in the clinical diagnostic process, and it may improve the quality and consistency of diagnoses; improve the accuracy of determining cancer susceptibility, recurrence and survival prediction; reduce the time required for diagnosis; prevent physical and psychological factors from influencing the diagnoses; and perhaps reduce hospital costs (38,39). Although AI-assisted diagnostic technologies have shown promise, no products related to AI-assisted CT diagnostic technologies for the classification of pulmonary nodules as benign or malignant have been approved by the National Medical Products Administration (NMPA) in China (40). Currently, few systematic reviews of the diagnostic performance of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant have been conducted, no systematic review has been performed to compare the diagnostic performance of this technology with different MLAs, and the benefits and risks of this technology perceived by physicians remain unclear.

This study aimed to systematically review the diagnostic performance of AI-assisted CT diagnostic technology in the classification of pulmonary nodules as benign or malignant and to analyze physicians’ perceptions of its potential benefits and risks as well as their attitude toward its clinical application in China.

We present the following article in accordance with the PRISMA reporting checklist (available at https://dx.doi.org/10.21037/jtd-21-810).

Methods

Literature search and eligible studies

To conduct a systematic review, 6 electronic literature databases [PubMed, EMBASE (Ovid SP), the Cochrane Central Register of Controlled Trials (the Cochrane Library), the China National Knowledge Infrastructure (CNKI), the Wanfang Data Knowledge Service Platform (WANFANG data), and the China Biomedicine Database (Sinomed)] were searched for eligible studies from 2010 to 2019. The MeSH terms or keywords used for literature retrieval were as follows: (pulmonary nodule OR lung nodule OR pulmonary neoplasm OR pulmonary cancer OR lung cancer OR lung tumor OR lung neoplasm) AND (artificial intelligence OR AI OR computer-aided diagnosis OR computer-assisted diagnosis OR CAD OR deep learning OR neural network OR machine learning OR support vector machine OR decision tree) AND (computed tomography OR CT) AND (classification OR classifier OR classify OR diagnosis). The search strategy was designed according to different database searching standards by two reviewers. Additionally, a manual search of the literature was conducted to further identify eligible studies.

After the literature search, each study related to the diagnostic performance of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was screened to determine whether it met the inclusion and exclusion criteria. The inclusion criteria were as follows: (I) the purpose of the study was to assess the diagnostic performance of AI-assisted CT diagnostic technology in the classification of pulmonary nodules as benign or malignant; (II) at least one machine learning algorithm was used as a classifier; (III) a CT imaging modality was used; (IV) sufficient data were provided in or could be calculated from the articles, including the number of true positives (TPs), false positives (FPs), true negatives (TNs) and false negatives (FNs); (V) the full text of the article was available; and (VI) the article was published from 2010 to 2019. The exclusion criteria were as follows: (I) case reports, conference abstracts, reviews, posters, or other nonoriginal articles; (II) duplicate publications or studies with overlapping sample data; and (III) articles not written in Chinese or English.

Data extraction

The quality of the included studies was assessed using the second version of the quality assessment of diagnostic accuracy studies (QUADAS-2) scale (41). The quality score of each study was determined on the basis of 14 items from four domains (description, signaling questions, risk of bias and concern about applicability). The specific data extracted from each study were the first author’s name, year of publication, first author’s country, data source, gold standard, machine learning algorithm used, number of nodules, and data used to evaluate diagnostic performance (TP, FP, TN and FN). Two reviewers carefully rated the quality of all the included studies and extracted the data from these studies independently in accordance with the study protocol. Disagreements were resolved by discussion between the two reviewers.

Moreover, to develop AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, data related to the texture features extracted from CT images are usually divided into a training dataset and a testing dataset. The training dataset is used to develop a model for pulmonary nodule classification, while the testing dataset is used to validate the model created using the training dataset. In our systematic review, the results obtained from testing datasets were used for the meta-analysis and meta-regression. Some studies did not specify the training dataset and testing dataset, and in this case, all samples were extracted (5 studies). If a study included more than one testing dataset to test the classifier, all results from the testing datasets were recorded and used for the analyses (3 studies). If a study evaluated the classification accuracy using the chi-square test at different confidence levels, only the result at 95% confidence level was recorded and used for the analyses (1 study on Bayes classifier). If a study used many MLTs to classify pulmonary nodules, all of them were mentioned and recorded in the data tables (6 studies).

Questionnaire survey

A cross-sectional questionnaire survey of physicians from 9 public tertiary hospitals in Shanghai, Hubei Province and Gansu Province in China was conducted from September to December 2019. These regions were selected to capture various socioeconomic statuses (high, middle and low) and geographic distributions (eastern, central and western) within China. Within each region, 3 public tertiary hospitals (1 general hospital and 2 specialty hospitals) were selected for the questionnaire survey.

Within each selected hospital, all physicians in the clinical departments and imaging departments related to the diagnosis and treatment of respiratory diseases and/or lung cancer (such as the departments of respiratory disease, thoracic surgery, oncology, and medical imaging) were invited to participate in the questionnaire survey. The survey of hospital physicians was conducted by anonymous, paper-based, self-administered questionnaires. The questionnaires were distributed and collected by hospital managers who were trained and served as coordinators of the study. In the training for the survey, the importance and aims of the study were provided.

The questionnaire included two sections: one section concerned the physician perceptions of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant (such as its benefits and risks and their personal attitude toward its clinical application), and the other section concerned general information on the physicians (age, sex, educational attainment, department in which they worked, whether they were physicians or physician managers, and their practical experience with the technology) (Appendix 1). The physicians’ personal attitudes toward the clinical application of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant were rated using a 5-point Likert scale (5= strongly supported, 4= somewhat supported, 3= neutral, 2= somewhat unsupported, 1= strongly unsupported). The percentage of physicians who gave scores equal to or greater than 4 regarding their attitude toward the clinical application of the technology was referred to as the support rate.

Data analysis

We analyzed the study characteristics, quality of the included studies, and characteristics of the surveyed physicians. The pooled sensitivity, specificity, positive likelihood ratio (PLR), negative likelihood ratio (NLR), diagnostic odds ratio (DOR), summary receiver operator characteristic (SROC) curve, and area under the SROC curve (AUC) with 95% confidence intervals (CIs) were calculated to assess the diagnostic performance of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant. AUC values of 0.90–1.00 indicate excellent detection, 0.80-0.90 indicate good detection, 0.70–0.80 indicate fair detection, 0.60–0.70 indicate poor detection and 0.50–0.60 indicate failure (42).

Statistical methods

The Spearman correlation coefficients of the logarithm sensitivity and 1-specificity were calculated to detect the threshold effect (43). The Cochran-Q test and I2 were used to assess the no-threshold effect. If there was no significant heterogeneity caused by the no-threshold effect (P<0.05 and I2≤50%), a fixed-effects model was adopted; otherwise, a random-effects model was used (44, 45). In addition, Deek’s asymmetry test was used to assess publication bias.

A multilevel linear regression model (method = REML, weight = 1/variance of odds) was used for the meta-regression to explore the effects of the machine algorithms on the pooled DOR while controlling for the study random effects and other fixed effects (number of nodules and the first author’s country). Using the above multilevel linear regression model, the adjusted pooled DORs of AI-assisted CT diagnostic technologies for the classification of pulmonary nodules as benign or malignant were calculated and graphed, with the number of nodules and the first author’s country at the means.

In addition, chi-square tests were used to compare the perceptions of different groups of physicians toward AI-assisted CT diagnostic technologies for the classification of pulmonary nodules as benign or malignant. A univariate logistic model was used to analyze the association of physicians’ perceptions of the benefits and risks of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant with their rate of support for its clinical application.

The data were analyzed by using Review Manager 5.3 (RevMan 5.3), Meta-Disc 1.4, Stata 12.0, and SAS 9.4. All P values were two-sided, and P<0.05 indicated statistical significance.

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Institutional Review Board of the School of Public Health, Fudan University (IRB#2019-07-0767), and oral informed consent was obtained from all surveyed individuals because the survey was anonymous and carried no more than minimum risk. The systematic review in the study was not registered.

Results

Diagnostic performance

Study selection

From the 6 selected electronic literature databases, 1,859 studies were retrieved by search strategies. After the titles and abstracts were reviewed and duplicate publications were removed on the basis of the inclusion and exclusion criteria, 545 studies were selected for the next step. The full texts of the articles were read and screened based on the inclusion and exclusion criteria, and 2 additional studies were identified from the references of the articles. Finally, 28 studies related to AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant were included in the systematic review ().

Figure 1

Flow chart of the search for eligible studies.

Study characteristics

Among the 28 studies on AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, which included 5,727 pulmonary nodules, 25.00% of the studies were published in 2018; 71.43% of the first authors were from China; 25.00% used the gold standard of diagnoses from the LIDC-IDRI dataset; 42.86% used the gold standard of pathologic diagnoses or follow-up; and 32.14%, 25.00% and 10.71% of the data used for machine learning were from hospitals, the Lung Image Database Consortium of Image Database Resource Initiative (LIDC-IDRI) dataset (46) for both hospitals and the LIDC-IDRI dataset, respectively (Appendix 2).

In the development of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, SVMs, DBNs, DTs, CNNs, ANNs, BNs, Fuzzy C-means (FCMs) and other machine learning classification algorithms accounted for 29.41%, 15.69%, 13.73%, 9.80%, 5.88%, 3.92%, 3.92% and 17.65% of the total 51 times that classification algorithms were used in the 28 studies (more than one algorithm was used in some studies) (Appendix 2 and Appendix 3).

Original study results

The systematic review showed that AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant using different classification algorithms had sensitivity rates ranging from 52.00% to 100.00%, specificity rates ranging from 34.69% to 100.00%, and accuracy rates ranging from 56.00% to 100.00% (Appendix 3).

Quality assessment, heterogeneity and publication bias

Quality assessment of the 28 included studies showed that the majority of studies fulfilled the criteria of the reference standard. For example, the reference standard was likely to correctly classify the target condition. Almost all studies had an unclear risk of bias in patient selection and index tests because whether the samples were included randomly or consecutively was not explained, and a threshold was prespecified in the studies. With respect to flow and timing, 9 studies had a high risk of bias, because 7 of them did not adopt the same reference standard, while 2 of them did not include all samples in the testing datasets ().

Figure 2

Quality assessment of the included studies using the QUADAS-2 tool. “+”: low risk; “-”: high risk; “?”: unclear risk.

Spearman correlation analysis showed that the Spearman correlation coefficient between sensitivity and 1-specificity was −0.61 (P<0.05). The Cochrane-Q test showed that Q=79.54, I2=97.49% (95% CI: 95.82–99.15%), and P<0.01. These results indicated that there was no heterogeneity in the diagnostic test results caused by the threshold effect, but there was heterogeneity in the diagnostic test results caused by the no-threshold effect. Therefore, a random-effects model was used in our meta-analysis.

Deek’s asymmetry test showed that the coefficient of bias was −17.85 with P<0.05, which indicated that there was potential publication bias.

Pooled diagnostic performance

The meta-analysis weighted by the number of nodules showed that the pooled sensitivity, specificity, PLR and NLR of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant were 0.90 (95% CI: 0.87–0.92), 0.89 (95% CI: 0.85–0.91), 7.95 (95% CI: 5.92–10.67), and 0.11 (95% CI: 0.09–0.15), respectively. The pooled DOR was 70.33 (95% CI: 41.39–119.51) ().

Table 1

Meta-analysis of AI-assisted CT diagnostic technology†

Indicators	Estimate (95% CI)	I2 (95% CI) (%)	Cochrane’s Q (P value)
Pooled sensitivity	0.90 (0.87, 0.92)	89.27 (86.99, 91.55)	466.13 (0.00)
Pooled specificity	0.89 (0.85, 0.91)	89.73 (87.58, 91.89)	486.91 (0.00)
Pooled positive likelihood ratio	7.95 (5.92, 10.67)	91.16 (91.16, 93.99)	673.47 (0.00)
Pooled negative likelihood ratio	0.11 (0.09, 0.15)	90.99 (89.17, 92.81)	555.09 (0.00)
Pooled diagnostic odds ratio	70.33 (41.39, 119.51)	100.00 (100.00, 100.00)	1.4e+55 (0.00)
Overall	–	97.49 (95.82, 99.15)	79.54 (0.00)

†, a random-effect model was used in the meta-analysis. AI, artificial intelligence.

The AUC of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was 0.95 (95% CI: 0.93–0.97) ().

Figure 3

SROC curve with confidence and predictive ellipses for AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant. SROC, summary receiver operator characteristic; AUC, area under the SROC curve; AI, artificial intelligence.

Meta-regression analysis

After a multilevel linear regression model was used to control for the study random effects and other fixed effects (number of nodules and countries), significant differences in the log values of the pooled DOR were detected between classification algorithms used in AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant. The log (pooled DOR) for DBN was not different from that for ANN, but was significantly higher than those for other types of classification algorithms. In addition, the model showed that the log (pooled DOR) in the studies of which the first authors were from China was nine times lower than that in studies with first authors from other countries (). The adjusted pooled DORs for DBN, CNN, DT, ANN, SVM, and other classifiers, as calculated using the above model, were 1318.57, 151.52, 127.46, 65.85, 54.96 and 60.95, respectively, and the adjusted pooled DORs in the groups in which the first authors were from China and from other countries were 47.71 and 372.56, respectively. All of the above adjusted pooled DORs were significantly higher than 1 (Figure S1).

Table 2

Meta-regression of the Log (pooled DOR) of AI-assisted CT diagnostic technology†

Variable	Estimate (95% CI)	Standard error	P value
Intercept	9.4338 (7.4538, 11.4137)	0.9818	<0.0001
Algorithms (control = deep belief network)
Support vector machine (1: yes, 0: no)	−3.1780 (−4.5967, −1.7587)	0.7036	<0.0001
Decision tree (1: yes, 1: no)	−2.3370 (−3.8411, −0.8320)	0.7460	0.0031
Convolutional neural networks (1: yes, 1: no)	−2.1640 (−3.8342, −0.4931)	0.8284	0.0124
Artificial neural network (1: yes, 0: no)	−2.9970 (−6.1486, 0.1548)	1.5628	0.0618
Others (1: yes, 0: no)	−3.0740 (−4.6794, −1.4692)	0.7959	0.0004
No. of nodules (1: ≥150, 0: <150)	−0.8420 (−1.8731, 0.1889)	0.5112	0.1068
China (1: yes, 0: no)	−2.0550 (−3.6124, −0.4980)	0.7722	0.0109

†, a multilevel linear regression model (method = REML, weight = 1/variance of odds) was used to control for the study random effects. DOR, diagnostic odds ratio; AI, artificial intelligence.

Physicians’ perceptions

Physician characteristics

Among 406 questionnaires sent to the selected physicians, 345 questionnaires were completed and returned. The response rate of the survey was 84.98%. Among the 345 physicians who responded to the survey, 32.46%, 38.84% and 28.70% were from Shanghai, Hubei Province and Gansu Province, respectively; 69.28% were from general hospitals; and 50.43% and 32.17% were from oncology and imaging departments, respectively. In total, 46.96%, 49.28%, 79.71% and 16.52% of the physicians were in the age group of 30–39 years, were male, had attained a master’s or PhD degree, and were physician managers in a clinical department, respectively. In addition, 20.87% of the surveyed physicians had practical experience using AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant (Appendix 4).

Perceptions of the benefits and risks

The study showed that 81.16% and 78.55% of the physicians perceived “reduced workload for radiologists” and “improved diagnostic efficiency”, respectively, as one of the top 3 benefits associated with AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant. Furthermore, 46.38% of the physicians perceived “high diagnostic accuracy” as one of the top 3 benefits. In addition, more physicians with practical experience in using AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant perceived “improved diagnostic efficiency” as one of the top 3 benefits (88.89%) compared to those without this experience (75.82%) ().

Table 3

Physicians’ perceptions of the benefits and risks of AI-assisted CT diagnostic technology

Items	Total (n=345)			With experience (n=72)			Without experience (n=273)		χ2
	No.	Percent (%)		No.	Percent (%)		No.	Percent (%)
Benefits (listed in the top 3)
High diagnostic accuracy	160	46.38		34	47.22		126	46.15	0.03
Improved diagnostic efficiency	271	78.55		64	88.89		207	75.82	5.77*
Reduced diagnostic expense	98	28.41		9	12.50		89	32.6	11.32***
Improved patient satisfaction	97	28.12		16	22.22		81	29.67	1.56
Reduced workload of radiologists	280	81.16		62	86.11		218	79.85	1.46
Reduced number of radiologists	136	39.42		17	23.61		119	43.59	9.52**
Risks (listed in the top 3)
Leakage of patient privacy	144	41.74		38	52.78		106	38.83	4.56*
Increased risk of misdiagnosis	202	58.55		43	59.72		159	58.24	0.05
Increased risk of missed diagnosis	154	44.64		25	34.72		129	47.25	3.62
Increased diagnostic expense	157	45.51		22	30.56		135	49.45	8.20**
Reduced diagnostic competence of radiologists	167	48.41		22	30.56		145	53.11	11.61***
Lack of a unified diagnostic standard	192	55.65		49	68.06		143	52.38	5.67*
Increased workload of radiologists	35	10.14		7	9.72		28	10.26	0.02

*P<0.05, **P<0.01, ***P<0.001. AI, artificial intelligence.

The study also showed that 58.55%, 55.65%, 48.41%, and 45.51% of the physicians perceived “increased risk of misdiagnosis”, “lack of unified diagnostic standard”, “reduced diagnostic competence of radiologists” and “increased diagnostic expense”, respectively, as one of the top 3 risks associated with AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant. A total of 41.74% of the physicians perceived “leakage of patient privacy” as one of the top 3 risks. In addition, more physicians with practical experience perceived “lack of unified diagnostic standard” and “leakage of patient privacy” as one of the top 3 risks compared to those without practical experience (68.06% vs. 52.38% and 52.78% vs. 38.83%, respectively), but fewer physicians with practical experience perceived “increased diagnostic expense” and “reduced diagnostic competence of radiologists” as one of the top 3 risks compared to those without practical experience (30.56% vs. 49.45% and 30.56% vs. 53.11%, respectively) ().

Attitude toward the clinical application of the technology and its influencing factors

The study revealed that the rate of physicians’ support for the clinical application of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was 73.62%. Physicians with practical experience had a significantly higher support rate than those without practical experience (87.50% vs. 69.96%, χ2=9.02, P<0.01), even after controlling for physician characteristics [age, sex, type of employee (physician vs. physician manager)] and the physicians’ perception of the benefits and risks associated with AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant ().

Table 4

Factors associated with supporting the clinical application of AI-assisted CT diagnostic technology for classification of pulmonary nodules†

Parameters	β	SE	χ2 Wald
Intercept	1.39	0.92	2.31
Age (years)	−0.01	0.02	0.65
Sex (1: male, 0: female)	−0.04	0.27	0.02
Physician manager (1: yes, 0:no)	−0.42	0.40	1.12
Practical experience (1: yes, 0: no)	1.03	0.40	6.49*
Benefits (1: yes, 0: no)
High diagnostic accuracy	0.25	0.31	0.67
Improved diagnostic efficiency	0.70	0.33	4.45*
Reduced workload of radiologists	0.07	0.36	0.04
Reduced number of radiologists	0.68	0.29	5.34*
Risks (1: yes, 0: no)
Increased risk of misdiagnosis	−0.72	0.32	5.11*
Increased diagnostic expense	−0.42	0.30	2.07
Reduced diagnostic competence of radiologists	−0.54	0.30	3.23
Lack of a unified diagnostic standard	0.13	0.28	0.21
−2 Log L			398.10
χ2 likelihood			30.24**

†, the data from 345 surveyed physicians were used for the logistic analysis. If the physicians strongly supported or somewhat supported the clinical application of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, the dependent variable in the model was coded as “1”; otherwise, it was coded as “0”. *P<0.05, **P<0.01. AI, artificial intelligence.

The univariate logistic model also showed that the physicians who perceived “high diagnostic accuracy” and “reduced number of radiologists” as one of the top 3 benefits had a higher support rate for the clinical application of the technology than did those who did not perceive these factors as a top benefit, while the physicians who perceived “increased risk of misdiagnosis” as one of the top 3 risks had a lower support rate than did those who did not perceive this factor as a top risk ().

Discussion

Good diagnostic performance

Our analyses showed that the quality of the included studies was generally acceptable and that the AUC of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was 0.95 (indicating a level of excellence). More specifically, the pooled sensitivity, specificity, PLR and NLR of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant were 0.90, 0.89, 7.95, and 0.11, respectively, and the pooled DOR was 70.33. Our meta-analysis confirms that in the context of lung cancer diagnosis, AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant has good diagnostic performance, although the pooled specificity for pulmonary nodule diagnosis (0.89, 95% CI: 0.85–0.91) needs to be further increased by improving the feature extraction methods and reducing the FP rate (47,48).

The trade-off between sensitivity and specificity is crucial for diagnostic technology. When sensitivity and specificity have an inverse relationship, this indicates that a threshold effect is causing heterogeneity in the diagnostic test results (49). The ROC curve also presents the trade-off between sensitivity and specificity. A summary operating point with a small confidence region that is positioned in the upper left corner supports a desirable diagnostic performance of a technology (50). Our study found that no threshold effect was introduced in the meta-analysis and that the summary operating point had a relatively small confidence region (sensitivity: 0.87–0.92 and specificity: 0.85–0.91, respectively) and was positioned in the upper left corner (AUC =0.95). These findings support the good diagnostic performance of this technology.

In addition, the SVM, CNN, ANN and DBN algorithms were most commonly used for diagnosis classification in the included studies. This finding might be either because they have been recently developed and have accurate predictive performances (SVM, CNN and DBN) or because they have been used extensively for nearly 30 years (ANN) (51-53). In particular, the DBN algorithm has witnessed success and shown promising prospects in the classification of pulmonary nodules as benign or malignant in recent years (54). Our meta-regression showed that after the study random effects and other fixed effects were controlled for, the log value of the pooled DOR for DBN was higher than those for SVM, DT, CNN and other types of classification algorithms used in AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant.

The study also found that the log (pooled DOR) in the studies of which the first authors came from China was much lower than that in the studies with first authors from other countries. This may be caused by many factors, such as database resources, gold standards, image preprocessing, image segmentation, and nodule features extracted and selected from lung CT images.

The potential risk of study methodological bias and publication bias in the included studies may affect the validity of the results from the systematic review. Therefore, it is worth investigating the perceptions of physicians toward AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, especially those of physicians with practical experience with this technology.

Highly perceived benefits of an improved diagnostic efficiency and reduced workload

Our study found that AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was applied in many tertiary hospitals and used by a portion of relevant physicians. In 9 surveyed hospitals, 8 hospitals had applied this technology, and among 345 physicians who responded to the survey, 20.87% of them had practical experience using the technology. With the increasing demand for CT diagnosis of lung cancer, this technology is expected to be further developed and distributed in hospitals. Regarding the benefits of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, “reduced workload for radiologists” and “improved diagnostic efficiency” were highly perceived by 81.16% and 78.55% of the physicians, respectively, as one of the top 3 benefits, especially by those with practical experience. This finding is concordant with the general concept of AI-assisted diagnostic technology. This may be because AI excels at recognizing complex patterns in images and thus offers the opportunity to transform image interpretation from a purely qualitative and subjective task to one that is quantifiable and effortlessly reproducible (55). Furthermore, AI highlights and presents regions with suspicious imaging characteristics to the radiologists. These benefits could contribute to a relatively high rate of physician support for the clinical application of AI-assisted CT diagnostic technology for pulmonary nodules (73.62%), especially from physicians with practical experience (87.50%).

More concerns about diagnostic accuracy and patient privacy

CT, which is a commonly used diagnostic tool, provides a large amount of information about a patient’s health. However, correctly interpreting the information is a major challenge for physicians. AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant is expected to help physicians detect suspicious lesions that are easily missed and subsequently classify the lesions, thus improving the accuracy of diagnosis (56-58). Although our meta-analysis indicated that AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant has a good diagnostic performance, fewer than 50% of the physicians perceived “high diagnostic accuracy” as one of the top 3 benefits, and more than 50% of the physicians perceived “increased risk of misdiagnosis” as one of the top 3 risks of this technology. Other concerns about this technology included AI-assisted diagnostic standards, the effects on radiologists’ competence and diagnostic expenses, missed diagnoses, and patient privacy protection.

Physicians provide the diagnoses, and each CT image analyzed by a computer is verified by diagnostic physicians (59). Thus, AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant will not replace physicians in making a final diagnosis, and the effects of this technology on diagnostic accuracy and radiologists’ competence should not be excessively considered.

Nevertheless, it should be noted that physicians with practical experience had more concerns about unified diagnostic standards and patient privacy protection in the development of the technology than did those without practical experience (68.06% vs. 52.38% and 52.78% vs. 38.83%, respectively). Because large amounts of CT data are rarely curated in terms of labeling, annotation, segmentation, quality assurance, or fitness for the problem at hand (55), it is very difficult to establish a unified diagnostic standard. However, standardized benchmarking is of particular importance in the medical field, especially given the multitude of imaging modalities and anatomic sites, as well as acquisition standards and hardware (60). In addition, larger and publicly available databases are needed for improved validation (60). During the collection and use of patient data, measures to protect patient privacy and sensitive health information should be undertaken.

Factors affecting the rate of physician support for the clinical application of the technology

The overall rate of physician support for the clinical application of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant was 73.62%. Physicians with practical experience had a significantly higher support rate than those without practical experience (87.50% vs. 69.96%), and the logistic model demonstrated a similar result. Moreover, the study showed that diagnostic accuracy (including misdiagnosis) and practical experience significantly influence physicians’ level of support for the clinical application of this technology. These findings indicate that AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant should be further improved to better meet physicians’ expectations.

To improve the diagnostic accuracy of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant, extracting and selecting the most discriminative features (such as the size, shape, spatial complexity, intensity patterns, and a range of other “texture” features and “radiomics” of pulmonary nodules), developing large available datasets to capture a sufficiently broad disease spectrum and exploring better classification algorithms are the key issues and require cooperation among physicians, radiologists, and computer technicians (12).

Study limitations

Our study had some limitations. First, there was a risk of study methodological bias and publication bias in the included studies, which may affect the results from our systematic review. Second, the studies included in the meta-analysis were published from 2010 to 2019 and written in English or Chinese, and only one study from 2019 was included in this study because other identified studies from 2019 contained insufficient data for the analyses. These restrictions may have led to relevant studies being missed and not included in the systematic review. Third, the surveyed physicians were from public tertiary hospitals in China, and therefore, the findings from the survey have limited generalizability. Considering that public tertiary hospitals are the main medical setting in which CT scans for lung cancer are performed and clinical trials on the diagnostic performance of AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant are conducted in China, our findings from the representative survey of hospital physicians may reflect physicians’ perceptions of the technology to some extent.

Conclusions

AI-assisted CT diagnostic technologies for the classification of pulmonary nodules as benign or malignant have shown great promise and are under development. Our study confirms that in the context of lung cancer diagnosis, AI-assisted CT diagnostic technology for the classification of pulmonary nodules as benign or malignant has a good diagnostic performance (pooled DOR=70.33). The majority of physicians in China who were surveyed believed that this technology can improve diagnostic efficiency and reduce the workload for radiologists. Moreover, diagnostic accuracy (including the misdiagnosis rate) and practical experience were significantly associated with whether physicians supported the clinical application of this technology. Therefore, the specificity of this technology needs to be improved.

The article’s supplementary files as