Dr. Answer AI for prostate cancer: Intention to use, expected effects, performance, and concerns of urologists.

Objectives: To efficiently implement artificial intelligence (AI) software for medical applications, it is crucial to understand the acceptance, expected effects, expected performance, and concerns of software users. In this study, we examine the acceptance and expectation of the Dr. Answer AI software for prostate cancer.
Methods: We conducted an online survey for urologists from August 13 to September 18, 2020. The target software is an AI-based clinical software called Dr. Answer AI software, used for prostate cancer diagnosis. We collected data from 86 urologists and conducted a basic statistical and multiple regression analysis using the R package.
Results: The compatibility was significantly associated with the intention to use the Dr. Answer AI software. The expected average accuracy for the software ranges from 86.91% to 87.51%, and the urologists perceived that the cloud method is suitable to introduce the software. The most desirable function of the software for the specialists is predicting the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis after radical prostatectomy. Finally, the primary concerns involved the cost, compatibility with existing systems, and obtaining accurate information from the software. Conclusions: Our results present an understanding of the acceptance, expected effects, expected performance, and concerns of software users. The results provide a guide to help AI software be properly developed and implemented in medical applications.

Introduction

In recent times, the field of medical artificial intelligence (AI) has advanced significantly with the development of new technology. AI model development and services are being utilized for diagnosis, treatment, and outcome prediction in urologic diseases.1, 2, 3, 4

In Korea, the largest AI software (SW) project called Dr. Answer began in 2018, led by the National IT Industry Promotion Agency., The Dr. Answer AI project has 26 participating universities, associate general and general hospitals, and 22 companies focusing on eight diseases, including cardio-cerebral vascular disease, heart disease, breast cancer, colon cancer, prostate cancer (PCa), dementia, epilepsy, and incurable childhood genetic diseases. The Dr. Answer project aims to secure AI learning medical data for AI SW development to improve the diagnosis and treatment of major diseases.

Currently, the most important issue in the field of medical AI in South Korea is the licensing and commercialization of AI SW. Efficient utilization of such AI SW in medical applications is of utmost importance. Hence, understanding the acceptance and expected effects of SW users needs to be considered.

PCa is a common cancer that mainly occurs in the elderly, one of the leading causes of death in men in the world., Efforts are being made to overcome it by applying AI technology. The Dr. Answer AI project has developed various AI-based SWs. Here, we examine the acceptance, expected effects, expected performance, and concerns of the Dr. Answer SW AI for PCa.

Material and methods

Respondents: urologists

We conducted an online survey for urologists from August 13 to September 18, 2020. The survey was conducted with 1,100 members of the Korean prostate society (http://www.theprostate.org). All participants completed an anonymous survey. We sent the questionnaire via e-mail twice during the survey period. Before the survey, an explanation of the Dr. Answer AI SW was presented to the respondents, including descriptive images and videos. The participant data were de-identified during analysis.

Dr. Answer AI SW for PCa

Among the 26 participating hospitals in the Dr. Answer AI project, we focused on the PROstate Medical Intelligence System Enterprise-Clinical, Imaging, and Pathology (PROMISE CLIP) project. The PROMISE CLIP project has four hospitals: Seoul St. Mary’s Hospital of the Catholic University, Seoul National University Bundang Hospital, Samsung Medical Center, and Asan Medical Center. There are three SW used for PCa diagnosis: SWs for clinical, pathology, and imaging.

Among three SWs, we focused on clinical SW. LifeSemantics Corp. developed the clinical SW for PCa (Fig. 1), which is a company in the digital health platform business using AI., The clinical SW screen is largely divided into a patient information section and a clinical outcome prediction section. In patient information, there are patient information and lab data such as Atypical Small Acinar Proliferation, biopsy score, bone scan, CT, Gleason score, MRI images, PIN, PSA, and TNM (tumor, node, and metastasis). The SW shows the change over time of the PSA in graph format. In the clinical outcome prediction function, the SW provide clinical outcome such as prediction probability of occurrence in TNM staging, extracapsular extension, seminal vesicle invasion, lymph node metastasis, and predicting biochemical recurrence (BCR) after radical prostatectomy (RP). The clinical SW could help patients with PCa and physicians make treatment decisions.

Fig. 1

Dr. Answer AI SW for PCa.

Factors definition

Based on existing literature, we used a total of five factors.

Intention to use refers to the degree to which a physician’s behavioral intention tends to use the Dr. Answer AI SW. The intention to use is scored with a five-point Likert scale rated as: (1) strongly disagree, (2) disagree, (3) neither agree nor disagree, (4) agree, and (5) strongly agree. There are four questions (Supplementary Table 1).

The four independent variables are as follows: perceived usefulness, responsibility, reimbursement, and compatibility. The perceived usefulness is defined as the finding predictor of technology in the healthcare industry.,, We modified the definition of the originally perceived usefulness based on the TAM model. In this study, the perceived usefulness refers to the degree to which a urologist believes that the use of AI SW would improve his or her cancer treatment outcomes and processes. There are five questions.

Responsibility refers to the degree to which potential adopters are aware of the risks and liabilities for problems that may arise from the use of AI SW. To implement new technologies in medical applications, assuming responsibility for problems is a crucial issue. In the case of public health care aids such as telemedicine, the government may take responsibility for such services. AI SW can be implemented suitably when medical professionals have a clear responsibility for using the SW, and there is no burden associated with it. There are four questions.

For implementing new technologies sustainably in medical applications, the cost effectiveness of usage must be considered. Previous studies have explored the introduction and usage of technology in medical applications., In this study, reimbursement refers to the degree to which a physician believes using the Dr. Answer AI SW will be rewarded by financial support or compensation. There are four questions.

Compatibility refers to the degree to which a technology is perceived by potential adopters as consistent with their current values, previous experience, and needs. Compatibility positively influences the acceptance of AI technology in the healthcare industry. Wu et al. demonstrated that compatibility has a positive relationship with the perceived usefulness, ease of use, and usage intention in mobile computing acceptance. There are four questions. All questionnaires were scored with a five-point Likert scale.

Statistical analysis

We conducted a basic statistical and multiple regression analysis using the R program (version 4_3.5.0). We used the “enter” method for inputting independent variables as the variable selection method.

Ethical statement

The study was performed in accordance with the Declaration of Helsinki and was approved by three Institutional Review Boards: Catholic University (IRB number: KC20RISI0400).

Results

Participant characteristics

A total of 86 urologists out of the 1,100 members of the Korean prostate society participated. The response rate was 7.82% (Table 1). Since there were no missing values, all response data were used for analysis. In the case of urology, there are submajors such as children, urination, and tumors, but submajors were not classified.

Table 1

Demographic results

Variables		Frequency	Percent
Gender	Male	80	93.0
	Female	6	7.0
Age	20–29 years	1	1.2
	30–39 years	19	22.1
	40–49 years	44	51.2
	Over 50 years	22	25.6
Position	Intern	2	2.3
	Clinical fellow	3	3.5
	Pay doctor	12	14.0
	Professor	51	59.3
	Opening doctor	18	20.9
Career	Under 10 years	11	12.8
	11–20 years	49	57.0
	21–30 years	22	25.6
	Over 31 years	4	4.7
Hospital type	Private hospital	19	22.1
	Associate general hospital∗	2	2.3
	University hospital	57	66.3
	General hospital	8	9.3
Hospital location	Seoul	23	26.7
	Busan	7	8.1
	Daegu	5	5.8
	Incheon	3	3.5
	Gwangju	5	5.8
	Ulsan	2	2.3
	Gyeonggi-do province	17	19.8
	Chungcheongbuk-do province	3	3.5
	Jeollabuk-do province	1	1.2
	Jeollanam-do province	4	4.7
	Gyeongsangbuk-do province	3	3.5
	Gyeongsangnam-do province	2	2.3
	Cheju	3	3.5
	No response	8	9.3
Experience with CDSS	Yes	10	11.6
	No	76	88.4
Total		86	100.0

∗Associate general hospital: hospital with several medical offices, smaller than a general hospital. CDSS, clinical decision support system.

Of the respondents, 93% were male urologists and 51.2% of in their 40s, and 59.3% were professors, 66.3% were working in university hospitals, and 22.1% were working in private hospitals. Further, 46.5% worked in Seoul and Gyeonggi-do, and 11.6% had experience in using the clinical decision support systems.

Factor influencing the intention to use Dr. Answer AI SW

Before multiple regression analysis, five variables were evaluated on the internal consistency reliability using Cronbach’s alpha (Supplementary Table 2).

The values for all constructs ranged between 0.677 and 0.863 (0.816 for intention, 0.863 for responsibility, 0.784 for compatibility, 0.78 for reimbursement, and 0.677 for perceived usefulness). The values for the three constructs were >0.7. Consequently, Cronbach’s alpha for all constructs was reliable.

We performed a principal component analysis with varimax rotation to test the construct validity. The cross-loadings were lower than the corresponding factor loadings. Five factors emerged with no-cross construct loadings above 0.50. The analysis also demonstrated convergent validity with factor loadings exceeding 0.50 for each construct. The results confirmed the existence of four factors with eigenvalues >1.0, which accounted for 71.194% of the total variance. The communality ranged between 0.529 and 0.817, with all items achieving 0.50 (Supplementary Table 2). These results confirm that the four constructs were distinct unidimensional scales. We used four valid constructs: four items for intention, three items for responsibility, three items for compatibility, three items for reimbursement, and three items for perceived usefulness (Supplementary Table 2).

We derived the factors using multiple regression analysis (Table 2). The compatibility (t = 5.564) was significantly associated with only the intention to use the Dr. Answer AI SW (F = 12.000). The coefficient of determination (R2) for this model was 0.372, indicating that 37.2% of the variation in the intention to use the Dr. Answer AI SW can be attributed to a single independent variable.

Table 2

Multiple regression analysis results

Independent variables	Non-standardized coefficients		Standardized coefficients	t value	Sig.	Collinearity statistics
	B	SE	β			Tolerance	VIF
Constant	3.182	1.928		1.651	0.103
Responsibility	0.125	0.073	0.160	1.707	0.092	0.887	1.128
Reimbursement	0.168	0.099	0.157	1.707	0.092	0.913	1.096
Compatibility	0.717	0.129	0.524	5.564	0.000∗∗∗	0.875	1.143
Perceived usefulness	0.079	0.118	0.065	0.674	0.502	0.841	1.189

SE, Standard error; VIF, variance inflation factor.

R2 (adjusted R2) = 0.372 (0.341), F change = 12.000, significance of F change = <0.001. ∗∗∗t0.001=3.291.

Expected accuracy for the Dr. Answer AI SW and SW introduction method

The questionnaire included questions about the expected accuracy of the Dr. Answer AI SW. The questionnaire of expected accuracy is self-written, and the range of the input value is specified as 50%–100%. The expected accuracy is 86.91% in predicting the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis after radical prostatectomy. The expected accuracy is 87.51% in predicting TNM staging. In addition, the expected accuracy is 86.76% in predicting BCR after RP (see Table 3).

Table 3

Expected accuracy

Expected accuracy	Predicting the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis	Predicting TNM staging	Predicting BCR
Mean	86.91%	87.51%	86.76%
Median	90%	90%	90%
Mode	90% (n = 32)	90% (n = 36)	80% (n = 28), 90% (n = 28)

The following presents the results on the introduction of the Dr. Answer AI SW; 58.1% of respondents said that the cloud method is suitable. However, 86.4% of the respondents answered that, in the case of the stand-alone method, the linkage method with the existing system is suitable as the introduction method (see Table 4).

Table 4

Appropriate introduction approach

Introduction approach		Frequency	Percentage
Introduction method	Stand-alone method	36	41.9
	Cloud method	50	58.1
Stand-alone method	Interworking with existing system	74	86.0
	Without interworking with existing system	12	14.0

Expected effect of the Dr. Answer AI SW

After setting the state before SW development to 100, a questionnaire was developed to record the expected effect on this basis. The diagnostic accuracy was expected to improve by 79.3% on average. The treatment outcome was expected to improve by 78.84% on average. After treatment, the outcome explanatory time was expected to decrease by 26.28% on average. The outpatient time is expected to decrease by 34.07% on average. The respondents expected a 36.05% reduction in working hours on average (see Table 5).

Table 5

Expected effects

Expected effects	Improved diagnostic accuracy	Improved Treatment outcome	Reduced outcome explanatory time	Reduced outpatient time	Reduced working hours
Mean	79.30%	78.84%	26.28%	34.07%	36.05%
Median	90%	80%	20%	20%	20%
Mode	100% (n = 28)	100% (n = 25)	10% (n = 20)	0% (n = 20)	20% (n = 23)

Expected function of the Dr. Answer AI SW

We asked the respondents to rank the functions to be incorporated in the Dr. Answer AI SW. Most urologists responded that it is necessary to predict the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis after surgery with the Dr. Answer SW. This was followed by TNM staging prediction, death prediction, BCR prediction, quality of life, and time-series data in order (see Fig. 2).

Fig. 2

Derived functions of the SW.

Other responses include active surveillance, functional outcome after RP (urinary incontinence rate, sexual dysfunction, urination disorder, etc.), survival rate, prognosis prediction without RP, survival time prediction without radical prostatectomy, remote metastasis prediction, post-treatment symptom prediction, treatment pain prediction, recurrence rate, and complications in treatment.

Concerns about introduction of Dr. Answer AI SW

Urologists were concerned about the cost of using SW and its compatibility with existing systems. There were high concerns about whether they could properly receive the desired information. In addition, urologists were concerned about the extent to which the Dr. Answer AI SW would aid in diagnosis (see Fig. 3).

Fig. 3

Concerns about introduction of Dr. Answer AI SW.

Discussion

We examined the acceptance, expected effects, expected performance, and concerns of urologists in using the Dr. Answer AI SW for PCa.

First, compatibility is the most important factor in introducing the Dr. Answer AI SW. Compatibility is also a concern for SW acceptance. Compatibility may or may not be an influencing factor depending on potential users, technologies, and fields of introduction. Previous studies have determined that compatibility has a positive relationship with the intention to use, although some studies did not.

Urologists preferred that the introduction of AI SW should be compatible with their current methods. It is important to develop AI SW that is fully compatible with current systems and treatment methods. Urologists preferred the cloud method of introducing SW (58.1%). In addition, in the case of the stand-alone method, compatibility with existing systems is considered important (86.4%). Therefore, it should be considered whether it will be well reflected in the introduction environment and treatment process from the beginning of SW development.

Second, the average expected accuracy for the SW ranged from 86.91% to 87.51%. The expected accuracy of predicting the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis after radical prostatectomy is 86.91%. The expected accuracy is 87.51% in the case of predicting TNM staging. In addition, the expected accuracy is 86.76% in case of BCR after RP. When developing medical SW algorithms, there are no guidelines for reference accuracy. Moreover, expectations for accuracy vary depending on the disease and the intended use of the SW. The expected accuracy determined in this study can be used as a reference when developing a similar SW.

Third, the diagnostic accuracy was expected to improve by 79.3%, and the treatment outcome was expected to improve by 78.84%. The effect that urologists expected was higher than our estimate. In addition, after treatment, the outcome explanatory time was expected to decrease by 26.28%. The outpatient time is expected to decrease by 34.07%. Respondents expected a 36.05% reduction in working hours. Long waiting time and appropriate outpatient time have been previously discussed and reported to increase patient satisfaction., However, reducing the outpatient time is difficult in healthcare. Depending on the characteristics, each department has a different treatment time to satisfy patients. Saving time by 26%–36% is a significant result. According to Kim et al., the average outpatient cost for Korean patients after RP was reported at $161 per year. Based on this result, the economic value of the reduced outpatient time can be calculated. If the outpatient time can be reduced by 34.07%, it can be effective at saving $54.74 per patient per year. As the outpatient time is reduced, the idle time can be increased for urologists, and they can potentially attend to more PCa patients. Thus, the patient waiting time can also be reduced. The cost reduction effect can also be used for medical cost analysis. According to Bennett and Hauser (2013), it is reported that more than 50% of improvement can be achieved at half the cost through AI algorithms. We hope that research on the effectiveness and cost reduction effect of AI will continue in the future.

However, it should be considered that there are many people who responded with 0% in the expected effect response. There is a need to develop SW that can alleviate these concerns of urologists. Various SW for PCa are being developed and commercialized, and these low expectations can also be expected to change.

Fourth, most urologists responded that the occurrence of extracapsular extension, seminal vesicle invasion, and lymph node metastasis should be predicted after surgery with the Dr. Answer SW function. Thereafter, TNM staging prediction, death prediction, BCR prediction, quality of life, and time-series data become in order. In the case of PCa, the mortality rate is low; thus, it is difficult to obtain data to develop a model that predicts mortality, which was not incorporated in the Dr. Answer AI SW for PCa.

There are other SW features that urologists expect, such as active surveillance, functional outcome after RP (urinary incontinence rate, sexual dysfunction, urination disorder, etc.), survival rate, prognosis prediction without RP, survival time prediction without radical prostatectomy, remote metastasis prediction, post-treatment symptom prediction, treatment pain prediction, and recurrence rate and complications in treatment. These functions need to be investigated and developed in the future.

Finally, urologists were concerned about the cost of using the SW. There were significant concerns about whether they could properly receive the desired information. Urologists were concerned to what extent the Dr. Answer AI SW would aid in patient diagnosis. The cost effectiveness in the commercialization of AI SW remains to be solved. In the case of government and hospital-led projects, the results will be different. In addition, concerns about accurate information and usefulness are expected to be overcome through further developments and applications.

There are some limitations. First, we used data from 86 Korean urologists. Owing to the sample size, the interpretation of the analysis is limited. The results will be more generalizable if more responses are included. Second, the target SW is a clinical SW for PCa. In the case of PCa, the expected effect or accuracy may be different for pathology and imaging. Further research should be conducted according to specific SW targets and potential users for better results. Third, the expected effect was viewed based on the average value, but it can be evaluated differently in terms of the mode or median. Therefore, the explanation and interpretation of the results should be considered. Fourth, this study was conducted with urologists who have not used the Dr. Answer AI SW. In addition, only 11.6% of respondents have CDSS experience. There may be limitations in our results.

We present guidelines on the acceptance, expected effects, expected performance, and concerns of urologists in using AI SW. This study could help develop and introduce AI software that better reflects the medical environment.

Conflicts of interest

Mi Jung Rho and Jihwan Park are married couples and are co-participating in the project. For the rest of the authors, there are no conflicts of interest to declare.