Paradigm shift: cognitive surgery.

In the last hundred years surgery has experienced a dramatic increase of scientific knowledge and innovation. The need to consider best available evidence and to apply technical innovations, such as minimally invasive approaches, challenges the surgeon both intellectually and manually. In order to overcome this challenge, computer scientists and surgeons within the interdisciplinary field of "cognitive surgery" explore and innovate new ways of data processing and management. This article gives a general overview of the topic and outlines selected pre-, intra- and postoperative applications. It explores the possibilities of new intelligent devices and software across the entire treatment process of patients ending in the consideration of an "Intelligent Hospital" or "Hospital 4.0", in which the borders between IT infrastructures, medical devices, medical personnel and patients are bridged by technology. Thereby, the "Hospital 4.0" is an intelligent system, which gives the right information, at the right time, at the right place to the individual stakeholder and thereby helps to decrease complications and improve clinical processes as well as patient outcome. ©2017 Kenngott H.G. et al., published by De Gruyter, Berlin/Boston.

Cognition and surgery

With its ability to collect, process and analyze, the brain sets humans apart from any other species. “Cognition” is the core essence of humans describing the collection of knowledge and the interpretation thereof with regard to a specific purpose and deriving a conclusion or action accordingly.

Performing surgery is among the surgical professional activities with the highest cognitive workload as it requires a broad set of senses and functions of the human brain: sight to visualize the anatomy, hearing and voicing for communication and leadership as well as somatosensory and motoric skills to touch and manipulate tissue. The surgeon plans the operation, comprehends the patient’s anatomy and acts accordingly during the operation in order to accomplish the overall goal. Madani et al. define five domains of intraoperative performance: declarative knowledge, personal resourcefulness, interpersonal skills, psychomotor skills and cognitive skills [1]. Surgical cognition therefore consists not only of correct instrument or manual handling, but also of profound knowledge of anatomy, pathology and the current literature regarding the best techniques and strategies. Usually, higher-level cognitive skills are implicitly required throughout the operative course and are gathered through repetitive experiences in surgical planning, intraoperative changes of strategy, error detection and prevention [2].

“Cognition-guided surgery”

The advances of information systems in the past few decades and their paramount impact have led to the term “fourth industrial revolution”. After the development of the steam engines in the 18th century, the introduction of divided labor and mass production in the 19th century and the automation and digitalization of production in the 20th century, we are now in the transition to complete cyber-physical systems, where different types of hardware and their users are fused together by a virtual network, which regulates and operates on its own [3].

“Cognition-guided surgery” in this context is a new interdisciplinary research field aiming at soft- and hardware-based solutions that assist the surgeon and physician in general. The overall vision is to develop and establish technical integrated, self-learning solutions that perceive information and perform a knowledge-based interpretation to provide context-aware assistance.

Historically surgery has always been crucial for the invention and development of new technical instruments to accomplish specific goals [4]. Traditionally, these instruments have assisted the surgeon in a specific technical task. Nowadays more and more medical devices with new functions are used in the perioperative and intraoperative process, e.g. sophisticated endoscopic cameras for advanced visualization such as fluorescence imaging and new adaptive electronic handheld instruments which are used for vessel sealing. All these systems have in common that the surgeon himself/herself still has to find and process all relevant information, interpret the current situation and perform a particular action manually generating a rising workload. New methods such as minimally invasive surgery or natural orifice transluminal endoscopic surgery demand even more extensive training and surgical abilities, especially in terms of cognitive skills. However, future manual instruments will assist the surgeon in this regard. “Cognition-guided surgery” entails new developments in information and cyber-physical systems. These systems and devices help surgeons with higher-level surgical skills, e.g. combining patient data with the patient’s anatomy and current surgical literature to develop the best surgical strategy, calling attention to risk structures and pitfalls during the operation and guiding the surgeon through challenging surgical steps. “Cognition-guided surgery” implies the modern form of surgery, which does not only entail passive, but more so intelligent and active devices (Figure 1).

Figure 1:

The principle of “cognition-guided surgery”. The system uses perception from intelligent devices and the “Intelligent Operating Room”. This information is interpreted using a surgical knowledge base in order to create a context-specific, context-aware action such as warning of risk structures or automatic camera guidance. By feeding its experience back into the knowledge base the systems learns.

Integrated operating room

Currently, there are several technological and scientific endeavors being made to realize the core of “cognitive surgery”: the operating room of the future (Figure 2). Today common operating rooms (Common OR) are still mostly seen as mere rooms in which operations are performed with on-demand usage of isolated medical devices. These devices are manually activated by either the surgeons themselves or the staff members. They also generate large streams of information that are collected and stored for maintenance purposes. However, the effective activation of the systems and the information they provide in relation to their medical use are still mainly interpreted manually by the operating teams based on their individual training and experience. The interpretation of most data in the operating room (OR) is realized through communication and recommendations of these human specialists. In addition, these bits of information of medical devices are dispersed and not integrated into a common platform. This is one of the reasons why poor quality of communication, factual knowledge or lack of experience and consecutive medical errors may occur [5]. The common integrated operating room (Common Integrated OR) however aims to integrate medical devices from multiple manufacturers and allows for aggregated information visualization and integrated steering. This operating room is today the general gold standard for new surgical buildings. The next evolutionary step in operating rooms is mostly a change of perspective. In the past medical devices were mere effectors, used for a certain technical task during operation. However, the activity or inactivity of a medical device and its parameters can also, especially when aggregated, be used as sensor information to identify, e.g. the surgical phase of the operation. This new interpretation of the Common Integrated OR as an operating room with medical devices used not only as effectors but also as sensors (Sensor OR) is the most imminent new technological step of operating rooms.

Figure 2:

Evolution of the “Intelligent Operation Room (OR)”. The scientific focus has steadily changed from mere connectivity and availability of patient information and imaging to knowledge modeling, intuitive control and autonomous assistance and guidance for the surgeon.

Current projects focus on different aspects in order to reach this goal. One aspect constitutes for the development of interfaces and protocols to connect different devices by different manufacturers. Another aspect is the development of platforms for integration of all devices so that data can be interpreted and proposals for specific actions can be made. As a prerequisite of cognitive surgery, the collaborative research project OR.NET has developed a communication standard for all devices in a specific OR, the Open Surgical Communication Protocol, that enables all devices to communicate in a common language and thereby interact with each other regardless of the manufacturer or the type of device [6]. Other projects are conducted in Japan and the USA. The Japanese “Smart Cyber Operating Theater” is a platform where intraoperative imaging and room cameras are used for robotic surgery and assistance in intraoperative decision making [7]. The American “Medical Device Plug-and-Play” interoperability program aims for the introduction of open standards of devices and health records beyond the surgical field.

Cognitive operating room

The cognitive operating room (Cognitive OR) aims not only to connect devices and collect a huge amount of data into a common infrastructure, but also to analyze, compare and automatically derive further conclusions from these data. The Cognitive OR is able to help the surgeon, e.g. by creating a recommendation for intraoperative or postoperative actions or by assistance during the operation, i.e. through instrument or camera guidance (Figure 3). To achieve this vision the respective surgical context has to be modeled in order to structure and analyze any medical device information based on the underlying clinical process step. Accordingly, mandatory technical knowledge representation can be achieved by ontologies, which describe all available objects, persons and interactions between them during an operation: e.g. anatomy, surgical strategy, devices and their possible usages. In the course of intraoperative events and chain of actions, complex process models are created, which define the significance of specific data during the operation.

Figure 3:

The concept of “Hospital 4.0”. All information about patients, caregivers and devices is connected through the whole operative process.

The Transregional Collaborative Research Center “Cognition-Guided Surgery” investigated several aspects of representing surgical knowledge: for example, LapOntoSPM was developed and validated in an experimental setting to detect distinct steps during an operation [8]. Several platforms have been developed for integration of surgical knowledge, device status and other patient information. For example, an information processing tool was developed as a clinical decision-support system to assist the surgeon in the treatment of liver patients [9]. The system searched the current literature, combined it with the actual patient data and guided the surgeon throughout the preoperative diagnostic process. The Watson Oncology System is similar in this regard and is already being implemented in several hospitals around the world [10]. For laparoscopy, autonomous camera guidance was developed, which analyzed intraoperative images, compared them to a defined surgical step with the help of an ontology, and learned the right camera position via feedback loop [9]. Several other approaches have demonstrated successful recognition of operative steps by means of analyzing output values of different devices, for instance during laparoscopic cholecystectomy [11], cataract operation [12] or brain surgery [13]. The majority of these surgical process models concern minimal invasive surgery [14], and most applications aim for intraoperative assistance using augmented reality or robotics [14]. Other projects have investigated algorithms to determine postoperative complications, such as infection with Clostridium difficile after colorectal surgery [15].

In summary, there are a growing number of scientific projects and industrial developments that aim to assist surgeons not only by performing a specific step during the operation, but also by accompanying, observing and advising them during the operation. Over the past few years, several commercial solutions of the “Integrated OR” have been proposed. However, this type of OR is still based on proprietary interfaces and concentrates on controlling these devices manually instead of creating autonomously guided systems. The so-called Sensor OR partly realizes this by integrating all available technical devices, so that the surgeon gains full control and gets real-time assistance (Figure 2). One of the higher level goals of “cognitive surgery” is to then enable the “Cognitive OR”, which, in addition, introduces surgical knowledge, patient data and higher-level data analyses into the “Sensor OR” to create a fully self-reliant assistant for the surgeon (Figure 2).

Cognitive surgery – the future of surgery

Surgery has developed from a pure manual craft to diverse specializations with highly specialized expert knowledge performing high-tech processes. The information load as well as the demand for device control and interpreting results will continuously rise. Therefore, we need to discuss the advancements in surgery not only in terms of economics, but also regarding patient safety, information transition between caregivers and difficult decision making among the growing number of therapy choices. Maier-Hein et al. state that “future advances in surgery will continue to be motivated by safety, effectiveness, and efficiency of care. The next paradigm shift will be from implicit to explicit models, from subjective to objective decision-making, and from qualitative to quantitative assessment. This will enable personalized treatment and will place patients and caregivers into the focus of future evolution” [16].

It is not enough to have laboratory parameters of a postoperative patient electronically available for each caregiver; they also have to be interpreted in the context of the operation and the patient’s current status. Current automatic alarm systems, such as for renal failure, were unsuccessful, because they underestimated the cognitive process and regarded too little information [17]. The solution will be to consider the whole patient process, from pre-hospital diagnostics, characteristics of the operation, postoperative course and follow-up (Figure 3). This network consists of the patient, caregivers and hospital devices and combines them to supply the relevant data. With this approach, the hospital of things or “Hospital 4.0” will consist of systems that guide surgeons through the operation planning, assist them during the operation, alarm them if postoperative complications occur and automatically arrange appointments for further follow-up.

Conclusion

The technologies and research areas that together compose “cognition-guided surgery” or “cognitive surgery” will have a strong impact on how medicine and surgery will be performed in our future hospitals and operating rooms. Challenges that have to be addressed are to develop better and more comprehensive methods to model medical knowledge, to improve system integration and communication (protocols), to implement new concepts for clinical data warehouses and centralized clinical decision support systems, to improve usability, to ensure appropriate data protection and security and to provide accordance with medical device directives. Although these challenges will be difficult to address and require strong interdisciplinary collaborations, the benefits are persuasive. “Cognitive surgery” will include a comprehensive view of patients, clinical information, surgical instruments and other devices in a hospital in order to indicate the appropriate therapy, enable precise operations with fewer complications and optimize patient management. The intention is to assist the surgeon not only during the operation, but also along the whole patient process by giving the relevant information at the right time and at the right place.

Click here for additional data file.