Explainable artificial intelligence (XAI): closing the gap between image analysis and navigation in complex invasive diagnostic procedures.

LITERATURE REVIEW: Cystoscopy is the gold standard for initial macroscopic assessments of the human urinary bladder to rule out (or diagnose) bladder cancer (BCa). Despite having guidelines, cystoscopic findings are diverse and often challenging to classify. The extent of the false negatives and false positives in cystoscopic diagnosis is currently unknown. We suspect that there is a certain degree of under-diagnosis (like the failure to detect malignant tumours) and over-diagnosis (e.g. sending the patient for unnecessary transurethral resection of bladder tumors with anesthesia) that put the patient at risk.
CONCLUSIONS: XAI robot-assisted cystoscopes would help to overcome the risks/flaws of conventional cystoscopy. Cystoscopy is considered a less life-threatening starting point for automation than open surgical procedures. Semi-autonomous cystoscopy requires standards and cystoscopy is a good procedure to establish a model that can then be exported/copied to other procedures of endoscopy and surgery. Standards also define the automation levels-an issue for medical product law. These cystoscopy skills do not give full autonomy to the machine, and represent a surgical parallel to 'Autonomous Driving' (where a standard requires a human supervisor to remain in the 'vehicle'). Here in robotic cystoscopy, a human supervisor remains bedside in the 'operating room' as a 'human-in-the-loop' in order to safeguard patients. The urologists will be able to delegate personal- and time-consuming cystoscopy to a specialised nurse. The result of automated diagnostic cystoscopy is a short video (with pre-processed photos from the video), which are then reviewed by the urologists at a more convenient time.

Introduction

AI robotics has enormous potential to assist health professionals in many daily tasks [1]. Especially, AI methods from the domain of the so-called deep learning [2]. These are deep neural networks that are very successful and can practically exceed human performance [3]. Unfortunately, these most successful models have a serious drawback. They are high dimensional, very non-linear, and therefore, very complex that they are no longer interpretable by human experts [4]. However, in medicine, transparency, re-traceability and explainability are essential criteria and even mandatory, both for legal reasons [5] and for ethical reasons [6-10]. It is often that the question of why is even more important than the actual AI result. The area of computer science that addresses the implementation of transparency, re-traceability and comprehensibility of such so-called black-box approaches is called explainable artificial intelligence (XAI). Currently, this is becoming more and more important, because the quality requirements of medical AI solutions are increasing accordingly and the human expert must be able to explain and understand why a machine decision has been made, and must not rely on the usual parameters such as accuracy and precision alone [11].

Translated to urology, one field of action would be an invasive diagnostic procedure such as cystoscopy. Cystoscopy is the Gold Standard for initial macroscopic assessments of the human urinary bladder to rule out (or diagnose) bladder cancer (BCa). Around 7% of all newly diagnosed cancers are BCa in the United States of America (USA), making it the fourth most common malignancy and among the top ten lethal cancers in men [12]. Only 4.7% of all newly diagnosed cancers are BCa in Germany, accounting for 3.2% of all cancer-related deaths [13].

Bladder cancer is the 10th most common form of cancer worldwide, with an estimated 549,000 new cases and 200,000 deaths in 2018 [14]. Bladder cancer is more common in men than in women, with respective incidence and mortality rates of 9.6 and 3.2 per 100,000 in men: about 4 times that of women globally. The disease ranks higher among men, in whom it is the 6th most common cancer and 9th leading cause of cancer death. For the Age-Standardised Rate per 100,000 population in 2018, Germany had a bladder cancer rate of 26.4 in men and a 6.3 in women [14].

Bladder cancer has two entities: non-muscle-invasive bladder cancer (NMIBC) and muscle-invasive bladder cancer (MIBC) (see Fig. 1). Its notably high recurrence rates range from 50 to 70% at 20 years, and put patients at risk for MIBC [15]. Periodic cystoscopic examinations thus remain the cornerstone for patients’ follow-ups to detect early recurrences and lower the risk of progression.

Fig. 1

Diagram of ‘bladder cancer staging’, starting with carcinoma in situ (CIS), then Ta, T1, T2, T3 and T4. Source: [16]

NMIBC represents 75% of primary diagnoses and is characterized by frequent recurrence [16]. Bladder cancer remains a heterogeneous disease with varying pathology, molecular background, diagnostic options, and subsequent therapies for the individual patient. The improvement of bladder cancer outcomes remains the goal of future research.

Although it is such an invasive procedure, cystoscopy of the urinary bladder plays a critical role in the diagnosis and follow-up of patients with bladder cancer. Non-invasive biomarkers like nuclear matrix protein 22 (NMP22) or human complement factor H-related protein (BTA TRAK) from urine (or others from blood samples) have failed to replace cystoscopy in early detection and follow-up [EAU guidelines on non-muscle-invasive bladder cancer (NMIBC)] [17]. In fact, they may only enhance the Gold Standard cystoscopy [18].

Molecular diagnostics are making rapid progress. Competition to AI could come in the form of non-invasive biomarkers such as Oncoprotein Keratin 17 (K17) [19]. K17 is highly sensitive and specific for detecting urogenital cancer recurrence in the urogenital tract during treatment follow up. However, new biomarkers like K17 still do not rule out cystoscopy, or rule in TURBT, overall.

However, conventional cystoscopy still has several shortcomings, as it is especially difficult to detect flat or small lesions like carcinomas in situ (Cis) via standard white light cystoscopy (WLC) [20]. The latest data suggest that early and higher recurrence rates after transurethral resection (TUR) for BCa may be due to lesions going undetected during initial cystoscopy [21, 22].

Several methods to enhance optical techniques were introduced in the last few decades, and most have proven to improve initial detection rates and operative outcomes after therapy [transurethral resection of bladder tumours (TUR-BT)] of BCa (see Fig. 2). For example, hexaminolevulinate (HAL) TUR-B [photodynamic diagnosis (PDD)], an adjunct to WLC which involves fluorescence to localise abnormal tissue. This method proved to achieve higher detection and better resection rates due to the enhanced visualisation of malignant lesions during TURBT than cystoscopy and white light resection alone [23].

Fig. 2

Kaplan–Meier estimates of 12-month recurrence-free survival stratified by treatment. WL TUR-B, n = 77, or HAL TUR-B (PDD), n = 68; P = 0.02. Source: [21]

In some countries, the vast majority of cystoscopies (e.g. over 90% in Germany) are performed with white light cystoscopy (WLC) which can make it difficult to view certain lesions. Different technical improvements also include blue-light cystoscopy (BLC) and narrow-band imaging (NBI) [24, 25] (see Fig. 3).

Fig. 3

Comparison of WLC with BLC and NBI. A Bladder cancer tumour viewed with WLC. B Same bladder cancer tumour as in A, but viewed with BLC (Cysview™), which reveals a lesion in pink against the BL (Source: [24]). C Ta cancer growth viewed with WLC. D Same Ta cancer growth as in C, but viewed with NBI (Source: [25])

The aforementioned technical improvements require a human professional to do the navigating and interpreting of what the cystoscopy reveals. The interpretation of such optical information would be one field for AI methods that could rely on convolutional neural networks (CNNs), machine learning (ML), or deep learning (DL) [26]. At present, the RaVeNNA-4p1 consortium is developing a digital platform with 4PI Endo-imaging for 3D-reconstruction, semantic segmentation and visualisation of the bladder for BCa.

The consortium focusses on the multi-class segmentation of cystoscopic images using DL. This enables the precise segmentation of classes of interest such as bladder tumours [26]. In addition, the RaVeNNA-4p1 consortium trained CNNs with confirmed images of bladder cancers vs. healthy urothelium. Their output revealed up to 93.0% sensitivity and up to 83.7% specificity differentiating between tumourous/malignant and healthy urothelium; the neural network was pre-trained by feeding it 1.2 million images from the ImageNet Dataset [27] (Imagenet: a large-scale hierarchical image database). Several CNN models were evaluated by Eminaga et al. who found that the Xception-based model achieved the highest accuracy score (99.52%) [28].

Besides optical and imaged-based advancements, all these introduced methods assume that the cystoscopy itself was a thorough scan. They also assume that every millimeter of urothelium in the bladder was recorded. At present, cystoscopic navigation is limited, especially for flexible cystoscopy. We are convinced that better navigation will lead to automation, and that automation will ensure more reproducibility—essential during follow-up for non-muscle-invasive bladder cancer patients.

Methods

The literature search was performed on the following databases: PubMed/MEDLINE, Scopus, dblp, SSRN, Academia.edu, ResearchGate, and eur-lex.europa.eu. The review focused on articles that were published between the years of 2007 and 2021. All available publications were analyzed and summarized herein after an interdisciplinary collaborative review process. The search used the following keywords: cystoscopy, blue-light cystoscopy, narrow-band imaging, endoscopic documentation, endoimaging, semantic segmentation, instance segmentation, visualization, non-invasive biomarkers, bladder cancer detection, bladder cancer diagnosis, explainable artificial intelligence, black box, robotic endoscopy, autonomous endoscopy, autonomous surgery, human rights, high-risk AI systems, fundamental rights, provider, user, conformity, compliance, accountability, responsibility, liability, human control.

Urethrocystoscopy for diagnosis and follow-up of bladder diseases

Since the German urologist Maximilian Nitze introduced cystoscopy to urology, it has improved continuously, driven by the need to enhance patient comfort and increase diagnostic accuracy (see Fig. 4). Urethrocystoscopy is one of the cornerstones for diagnosis and the follow-up of patients with urinary bladder disorders [15, 17]. Cytoscopic findings significantly help us explain the cause of symptoms and draw correlations with different bladder illnesses or systematic diseases. These patients may have visited the urologist after they noticed haematuria (i.e., blood in the urine).

Fig. 4

Timeline for the R&D of MIBC and NMIBC. Source: [16]. Diagram abbreviations: TNM stands for Tumour, Node, and Metastasis. BCG stands for Bacillus Calmette–Guèrin (a type of immunotherapy drug). FGFR3 stands for fibroblast growth factor receptor 3 (a protein coding gene). TUR-BT stands for transurethral resection of bladder tumours

Being so essential to detecting and identifying bladder tumours, guidelines have been made available outlining how a cystoscopy is performed and documented [15, 17]. These guidelines help avoid the severe consequences of failing to identify malignant lesions. Indeed, in follow-up, patients must undergo frequent cystoscopies (e.g. every 3 months for a 2-year period in high-risk tumour cases). About a million cystoscopies are performed every year in the USA alone [29].

In the surveillance setting, the enhanced flexible cystoscope [Federal Drug Administration (FDA) approved in 2018] revealed improved detection rates. However, it remains an invasive and sometimes painful procedure [30]. Several distraction methods have been studied (e.g. hand-holding, playing music, speaking, etc.) in an attempt to minimise patient discomfort and ultimately increase acceptance rates, especially for follow-up examinations of non-muscle invasive bladder cancer (NMIBC) [31]. This also illustrates the need for a more comfortable procedure, as this procedure must be repeated frequently with regard to NMIBC, to detect recurrence and progression [32].

Cystoscopy’s technical situation

Despite having guidelines [15, 17], cystoscopic findings are diverse and often challenging to classify [28]. Arriving at an accurate diagnosis is still largely dependent on the examiner’s experience, and the inter-observer variability is therefore wide [33]. Often in training, some spots on the bladder wall go undetected because of improper manipulation, especially during flexible cystoscopy. Since cystoscopy began, the difficulty of training junior urologists on how to use a cystoscope is a major point, as visually identifying a tumour within the bladder is so challenging and often imprecise because of various patient–surgeon- or cancer-specific factors [34]. Cystoscopy is usually performed as an outpatient procedure with the patient awake during the procedure. Patients often exhibit low tolerance for this procedure, and their tolerance correlates with the urologist’s conduct during the cystoscopy, be the cystoscope flexible or rigid. The cystoscope is rotated slowly to inspect the bladder sidewalls and bladder dome. The urologist can also apply manual pressure to the lower abdominal wall, just superior to the pubic bone, to visualize the anterior bladder neck and dome.

After lubrication, a rigid or flexible endoscope of 16–25 Fr is inserted via the female or male urethra into the bladder. Rigid telescopes with different angles (from 0° to 30° to 70°) can be used. Rigid cystoscopy is an often painful procedure, especially for male patients (as they have a longer urethra and/or enlarged prostate gland), which is why flexible cystoscopy (first introduced by Tsuchida and Sugaware in 1973) [35] is recommended for males [36].

Since the introduction of flexible cystoscopy, comparable to flexible scopes in endoscopy, the complexity of handling this instrument has increased. The flexible cystoscope enables the urologist to navigate throughout the entire bladder. Its orientation inside the bladder during the procedure is made harder and more complex because of the limited optical information provided. That in turn means the procedure lasts longer, leading to more discomfort for the patient. The American Cancer Society states that the average cystoscopic procedure itself takes 5–10 min; however, it can often take as long as 10–20 min depending on the patient’s history [37]. In addition to being complex and time-consuming, the procedure can be more mental and physical workload for the urologist. It is therefore important to feel confident with the instrument in order to minimise the diagnostic procedure’s duration without risking the failure to detect bladder lesions requiring surgical treatment.

The learning curve of flexible cystoscopy is quite long (i.e. an average of 122 flexible cystoscopies must have been done to attain an acceptable expertise level for a novice urologist) [38]. Flexible cystoscopies done by an inadequately trained trainee are usually considerably longer than rigid cystoscopies. First, the technical handling of a flexible cystoscope must be learned. Second, the urologist’s orientation, once inside the bladder, is somewhat difficult to maintain in some situations. Losing one’s orientation means pulling the cystoscope back and having to re-insert it, making this one of the most painful moments, especially for male patients; retraction can also lead to bleeding in the bladder neck, thus impeding an accurate visualisation of the bladder mucosa.

Finally, this procedure is usually carried out in an outpatient setting with the patient fully awake. Nevertheless, this intervention is crucial in the follow-up and primary diagnosis of bladder cancer, especially in the early stages, making it vital for the patient to receive a thorough, accurate diagnosis. Even with the recent advantages in local anaesthesia and enhanced visualisation using visual chips instead of fiber optics, the procedure is still uncomfortable for many patients—a major concern in light of patients’ reluctance, especially in cancer follow-up investigations [35, 39].

Simulators for urological procedures have been introduced to overcome the limited teaching/training time and practical operating room experience of residents/fellows [40], as well as increasing healthcare demands, fiscal constraints, and medico-legal considerations. Several types of simulators have been assessed. Their capability to enhance the training of residents/fellows has been widely successful [41, 42]. However, many of these simulators are still not employed during training. Some complain that they are too expensive/pricey. Also, novices have reported that they had trouble successfully relating the computer display to actual conditions encountered during cystoscopy—a factor raising the risk of error and longer learning curves [43].

Solution: dedicated diagnostic robots in Urology

A combination of robotic cystoscopy and XAI-enhanced imaging would overcome the manoeuvring and visual diagnostic issues encountered with flexible cystoscopy. Here the visualization component is important because it is here that precise human-AI interaction is sought, and for this, the international research community in the domain of visualization has developed a number of methods to improve the interactive processes, especially for tight interaction [44]. Such a combination would raise the standards of bladder cancer diagnosis and increase patient comfort. It would not require much experience to operate (i.e., a urologist is not required to operate it), and it could shorten the duration (e.g. 2–5 min instead of 10–20 min) of the procedure and improve diagnostic accuracy.

However, some patients have expressed worry about an invasive robotic procedure. Since the introduction of the DaVinci® tele-manipulator to urology, the term “robot” is commonly used in the literature and urology practice [45]. In everyday practice, it is sometimes hard to make the patient understand that it is not an autonomous robot alone performing the surgical procedure, but instead actually the urologist who steers the DaVinci® tele-manipulator. Most patients are relieved to learn that the “robot” is actually steered by a human and that it cannot make a single movement on its own. Acceptance and confidence in such a system must be established during the consultation before the patient gives informed-consent to this procedure.

In summary, an XAI robot-assisted flexible cystoscope could be inserted via the urethra into a patient’s bladder like a soft transurethral catheter (an 8–12 Fr flexible scope instead of a 16–25 Fr rigid scope). A trained nurse could insert it following the prescribed standard procedure for transurethral catheterisation. The urethra is anaesthetised by a local lubricant gel, which is already standard care. The nurse could stay bedside during the entire robotic procedure, and take out the cystoscope when the scan is complete.

Once inserted, a flexible XAI robot-assisted cystoscope would be able to semi-autonomously perform a quicker, more comfortable and standardised diagnostic scan of the entire urinary bladder. The major anatomic landmarks can be identified. The cystoscope can fill (to a specific amount) and empty the bladder to maintain a perfectly clear view. If the patients feel any discomfort, they could also interrupt the procedure by pressing an emergency button (patient-in-the-loop) or simply tell the nurse to stop the procedure. After the entire bladder is scanned in a standardised mode, the nurse retracts/pulls out the cystoscope, and the patient is discharged. The collected data (scans) can then be processed by AI and presented in a highly standardised mode to the urologist for the definitive diagnosis.

Urologists working in an outpatient setting would benefit from a quick and thorough performed diagnostic scan. Automated cystoscopy has the potential to increase the inter-observer and intra-observational comparability and lead to more efficient diagnostics, which would ultimately improve patient safety. An XAI system that provides interpretable-explanations would also subsequently increase trust among urologists, nurses, and patients. Cystoscopy is considered a less life-threatening starting point for automation than open surgical procedures. Moreover, by not having to perform a diagnostic cystoscopy, urologists would reduce their mental and physical workload, thereby lowering the risk of adverse events/incidents when carrying out actual life-threatening surgeries. Insurance providers are also interested in new AI solutions that lower costs (i.e., the cost of having a nurse/technician attend a procedure is lower than paying a urologist to attend). The expected outcome would also be enabled by the AI-enhanced cystoscopy’s professional second “opinion” that more accurately predicts a bladder cancer’s recurrence, and would thus avoid the unnecessary transurethral resection of bladder tumors (TUR-BT) as well as increase patient safety [46-48]. AI should be at least as accurate as an experienced urologist in recognizing and differentiating bladder lesions, thereby improving the reproducibility of cystoscopy. AI can also play an important role in education for residency training.

One central problem in cystoscopy is the lack of a standardized documentation [49] and descriptions for the lesions found inside the bladder (e.g. EAU Guidelines [15, 17] are limited). Waldbillig et al. found that in surveys of German and international urologists, only 1 in 5 rate their cystoscopy as complete (missing about 10% of bladder tissue during cystoscopy in outpatient setting) and a significant gap in information transfer from outpatient to clinic due to documentation that is only analog and descriptive, i.e., no standardized video or picture-based documentation in 52% of the cases [50]. For the first time, AI will offer a structured reporting system and solution for risk profiling of BCa.

Considerations for the practical use of AI solutions as medical products

AI medical devices are challenging from a regulatory perspective. We are experiencing a situation where algorithms that are still learning in the treatment process are not legally permitted. Before approval, the methodology must, therefore, be presented as a design freeze.

There are fundamental differences between medical technology markets in the USA and Europe. The FDA has already approved a significant number of AI applications in the medical field. As there were no products appropriate for the simplified 510 (k) approval process, many manufacturers submitted applications for a de novo procedure [51]. This resulted in most products being assigned to the second-highest Risk Class II. The subsequent path went through the so-called “Breakthrough Device” status. All these approvals were made with already frozen algorithms.

In the EU Regulation 2017/745, most such applications in Europe are assigned to the third-highest Risk Class IIa. For the manufacturer, this means that in addition to risk management according to ISO 14971, software-specific standards IEC 62304 and 82304-1 and their associated standards must also be complied with. In this respect, AI-based medical devices or medical devices with AI software elements are currently requiring significant efforts for granting approval.

The FDA has signalled new directions for future regulatory-logical processes in a discussion paper from 2019 [52]. Here, approvals for continuously learning solutions were held out in prospect. They would only be associated with a comprehensive quality management system. The problem here, however, is that the responsibility for the performance of such products is transferred to the manufacturers, even in light of the future’s unpredictability. This in turn can lead to scepticism and caution on industry’s part. For this reason, manufacturers should not only ensure quality management on an ongoing basis, they must also ensure continuous validation steps based on robust clinical parameters. Similar political movements are not yet foreseeable in Europe. However, we expect them in the near future. In general, AI-based medical devices are a key technological contribution to improving patient care, and they are certain to eventually become an integral part of everyday clinical practice.

The importance of XAI for robotic cystoscopy and its legal and ethical considerations

The European Union is very active in the field of ethics and in protecting human rights in AI. Its strategy is to ensure citizens’ trust in AI. This strategy places the principle of explainability at the center of its concerns, along with other essential values (auditability, traceability, transparency, accountability, etc.). In the 20 October 2020 [53] resolution on a framework of ethical aspects of artificial intelligence, robotics and related technologies, the European Parliament aims to enshrine in law the common ethical principles of AI issues to make them more effective (point Y). This resolution mentions explainability in paragraph 23. First, it asserts that the adoption of these principles helps to ensure citizens’ trust in those technologies. Second, it recognizes, however, that the degree of explainability is relative to the complexity of the concerned technologies and admits that it is not always possible to explain why a model has led to a particular result or decision, in particular for black box algorithms. Third, it concludes that abiding by these principles is an essential precondition for guaranteeing accountability. However, a European resolution has no binding force on Member States.

The European Union has gone further by developing its first legal framework in its 21 April 2021 [54] proposal for a regulation on artificial intelligence (AI Act). Contrary to a resolution, a European regulation, once adopted, shall apply in all Member States. This future regulation adopts a risk-based approach to AI systems, which is also adopted in the resolution on a framework of ethical aspects [53]. The AI Act is intended to regulate the professional uses of AI. When placing AI systems on the market and using them, it imposes obligations that are proportional to the level of risk. High-risk AI systems for natural persons’ health and safety or European fundamental rights are the subject of reinforced supervision. The AI Act proposes two classification rules for high-risk AI systems (Article 6). Either the AI system is listed in Annex III of the AI Act, establishing a list of automatically high-risk AI, but it does not include AI systems in health, or the AI system meets the conditions of Article 6, paragraph 1. In this case, it shall be considered high risk if this AI system constitutes a safety component of a product or a product covered by Annex II and is required to undergo a third-party conformity assessment with a view to its placing on the market.

As EU Regulation 2017/745 on medical devices is listed in Annex II and an AI robot-assisted cystoscopy requires third-party conformity assessment, this AI system should therefore be classified as high-risk. Strict obligations are placed on the providers of the high-risk AI system, such as responsibility for their compliance with the AI Act, drawing-up the technical documentation, keeping logs automatically generated by their systems or affixing the CE marking to their high-risk AI systems (Articles 16 to 23).

Even when a high-risk AI system is related to products, such as a robot, the robot manufacturer must fulfill the same obligations as the provider (Article 24). The future regulation is also aimed at high-risk AI systems professional users. It therefore concerns the doctor who implements an AI robot-assisted flexible cystoscope or the establishment the doctor belongs to. The professional user shall comply with a series of obligations (Article 29), which are more or less difficult to apply in practice. For example, using these systems in accordance with the instructions for use accompanying the systems, implementing the human oversight measures indicated by the provider, exercising control over the input data, monitoring the AI system operation, keeping the automatically generated logs, or interrupting the use of the AI system in the event of doubt as to its conformity. Non-compliance of the AI system with these requirements or obligations shall be subject to administrative fines of up to €20,000,000 or, if the offender is a company, up to 4% of its total worldwide annual turnover for the preceding financial year, whichever is higher (Article 71, paragraph 4).

The resolution on a framework of ethical aspects [53] also calls for important outcomes concerning the expected degree of explainability. Indeed, the latter “should depend on the context of those technical processes, and on the severity of the consequences of an erroneous or inaccurate output, and needs to be sufficient for challenging them and for seeking redress” (paragraph 19 of the proposal for a regulation contained in this resolution). In terms of AI robot-assisted cystoscopy, the doctor should not only need to provide information regarding the medical exploration and its risks, but also all the explanations concerning AI robotics so that the patient understands how the technology works, what the advantages and disadvantages are, or even the limits of the results obtained by the AI [55].

Consequently, the most relevant driver for XAI research is the growing relevance of legal aspects along with AI-assisted decision-making [6, 56]. Ultimately, it is mandatory in these areas that responsibility remain on the medical doctors, hence keeping the human-in-control [57]. Human control over the AI system is a central principle associated with responsibility. This is the case with the Montreal Declaration for the Responsible Development of Artificial Intelligence of 2018, which asserts that “only human beings can be held responsible for decisions stemming from recommendations made by AIS (Artificial Intelligence Systems), and the actions that proceed therefrom” (paragraph 9.1). The European Union has adopted a similar principle in its AI Act [54]. This proposal outlines that high-risk AI systems “shall be designed and developed in such a way, including with appropriate human–machine interface tools, that they can be effectively overseen by natural persons during the period in which the AI system is in use” (Article 14, paragraph 1). The provider shall then identify measures to ensure human control, build them into the AI systems before they are placed on the market when technically feasible, or entrust their implementation to the user (Article 14, paragraph 3). The doctors, or their establishment, therefore have a real role to play in the implementation of the human control principle.

It is important to add that the 20 October 2020 European Parliament resolution with recommendations to the Commission on a civil liability regime for artificial intelligence [58] puts in place a strict liability for high-risk AI systems (Article 4).

Future work

This new XAI robot-assisted cystoscopy solution holds the potential to not only detect cancer, but to detect benign lesions as well. This capability is currently missing in biomarkers. However, it is still unclear how cost effective the new XAI solution will be. Nevertheless, comparative studies using simulation analysis models (e.g. Monte Carlo [59]) could be carried out to mathematically build a case for the cost effectiveness of this new XAI solution.

Conclusion

Visual diagnostic issues in cystoscopy can occur during live operations when the urologist is distracted or over focused on wielding and navigating the cystoscope in the bladder (i.e., the urologist fails to detect a malignant lesion). The development of miniaturised (reduced to 8–12 Fr) XAI robot-assisted cystoscopes will improve the performance of flexible cystoscopy in terms of technical and diagnostic parameters. So far, there are still no biomarkers available that can compare to the accuracy of cystoscopy. Nevertheless, non-invasive biomarkers (e.g. K17) could still be used to stratify patients on waiting lists, to prioritise cases for XAI robot-assisted cystoscopy and avoid unnecessary hospital visits for other patients during a pandemic (e.g. COVID‑19) [60].

The following two points highlight the uniqueness of the XAI solution: (i) this novel technology will detect and reduce cases of false negatives and false positives in this diagnostic procedure; and (ii) it gives the patient (in-the-loop) more control than ever before during this diagnostic procedure. This semi-autonomous procedure will be controlled by the nurse as well. This is the best way to guarantee and maximise patient safety. This new solution has the potential to dramatically increase patient acceptance and adherence, making cystoscopy a much less dreaded procedure. With the introduction of dedicated evaluation robots, patients will benefit from a more comfortable, quicker, thorough, and effective diagnostic scan. However, the definitive decision to perform another transurethral resection of bladder tumors (TURBT) or schedule a follow-up cystoscopy will be made by the urologist. But with XAI, that decision will rest on a more solid diagnostic foundation.