Applicability of Augmented Reality in an Organ Transplantation.

Augmented reality (AR) delivers virtual information or some of its elements to the real world. This technology, which has been used primarily for entertainment and military applications, has vigorously entered medicine, especially in radiology and surgery, yet has never been used in organ transplantation. AR could be useful in training transplant surgeons, promoting organ donations, graft retrieval and allocation, and microscopic diagnosis of rejection, treatment of complications, and post-transplantation neoplasms. The availability of AR display tools such as Smartphone screens and head-mounted goggles, accessibility of software for automated image segmentation and 3-dimensional reconstruction, and algorithms allowing registration, make augmented reality an attractive tool for surgery including transplantation. The shortage of hospital IT specialists and insufficient investments from medical equipment manufacturers into the development of AR technology remain the most significant obstacles in its broader application.

Background

Augmented reality (AR) and virtual reality (VR) are fast-developing technologies that will revolutionize most of the branches of contemporary medicine, from neurosurgery to teaching. AR delivers virtual information or its elements to the real world, like in the Pokemon Go game, where, with a smartphone in your hand, you could seek out digital characters in a real neighborhood. Today more than 200 AR applications and platforms are offered for Android devices. Virtual reality (VR) is just the opposite: it immerses a user in a completely digital world shutting out the physical environment almost entirely. HTC Vive or Samsung VR Gear goggles are popular devices able to transport anyone to the virtual world. The newest technology is mixed reality, which combines both AR and VR features so that the digital and real-world objects interact. Microsoft HoloLens 2 is probably the most advanced prototype in this field; its practical applications for medicine are still under development with Philips Azurion technology for image-guided minimally invasive surgery.

AR in General Surgery

The computer systems used currently in surgery improve planning, help intraoperative navigation, and enhance postoperative control [1,2]. Utilization of diagnostic imaging in preoperative planning allows 3-dimensional (3D) visualization of the anatomical structures, including blood vessels and tumors, and designing planes and axes. In the most advanced imaging systems, the development of the digital avatars for the specific surgical instruments will allow predicting their usefulness in the given operating environment. Preoperative digital images must be segmented, i.e., divided into the graphically homogeneous areas representing distinct anatomical structures [3]. However, planning cannot be finished without the surgeon, who identifies important anatomical structures and overlaid graphic elements. Intraoperative navigation usually transports preoperative planning results into the operating room (OR). This step, however, requires automated, often real-time registration; i.e., transfer of different sets of data, for instance data coming from different diagnostic studies, into a unique coordinate system of the segmented diagnostic images, into real anatomy of the patient. Actual patient’s body video image and 3D-rendered reconstruction are synchronized using fixed markers such as the umbilicus, easy palpable iliac spine, or the costal edge [4]. A significant obstacle in the registration of preoperative imaging is the movement and deformation of soft organs resulting from the surgical access, respiration, heartbeat, surgical manipulation, and separation of tissues [5,6]. Construction of the biomechanical models of tissue deformations or repeated intraoperative imaging can solve this problem. Registration of 2 different images may bring a systematic error, though registration accuracy today is very high, with an average error lesser than 1 mm [7].

The image is usually projected on the monitor screen, and the actual operating field is not always visible. An operator continually has to change eye focus, from the screen to the operating field and back, and use their own spatial imagination to connect both pictures. A viewpoint of the computer-generated images is usually static and differs from that of an operator. Change of viewpoint of the projected images requires third-party assistance or an interface located outside an operating field. Dynamic adjustment of the viewpoint direction is a very desirable feature of the final system design. These limitations are particularly important during intraoperative 2D diagnostics (classical x-ray radiology and ultrasonography), which requires extremely accomplished spatial orientation and eye-hand coordination. This takes a lot of training and the learning curve is long. When real-time registration problems can be resolved, the digital, 3D, real-looking image of the operating field and anatomy will be available with tracking movements of the surgical instruments. Another problem to be resolved during design of the new solutions for AR is the need for fast discovery of the errors such as an incorrect registration and wrong tool tracking. This is very difficult when the physical operating field is not visible to a surgeon. Finally, the time it takes to complete the registration and mathematical conversions and to navigate correctly is also important. They should not significantly delay the projection of AR (the changes of the view angle and the alignment of the instruments should be registered in as short a time as possible). The time required for the procedure to be performed, usually under general anesthesia, must be as short as possible.

Current status of AR

Contemporary systems for AR can be grouped into 5 major categories [8].

Devices for augmented medical visualization accommodating video camera or transparent screen mounted on the monitor of the fluoroscope, ultrasonography, or computed tomography (CT) scanner. Such a solution allows for the adjustment and fine-tuning of diagnostic images with reality. A viewpoint from the diagnostic device, 2D projection of the image, and inability to use this method with available preoperative diagnostic studies (CT or magnetic resonance imaging [MRI]) are serious limitations. The major advantages are the simplicity of the systems and no need for additional calibration [9-12]. The simplest example of this modality, which has been applied in vascular surgery, is marking on a monitor screen an angiography-produced contour of the aorta and its major branches, and then anchoring a radiopaque stent or stent graft in a proper position guided by the marks drawn on the monitor screen.

Augmented optical devices can provide graphic information, for example, information retrieved from virtual CT colonography is added to a real-time video image from the microscope or endoscope. The technology allows for in situ visualizations close to clinical practice but can only be applied in situations where the optical devices are routinely used. It also requires additional calibration, and a narrow field defined by the optical system of the instrument [13,14]. A newer example of the application of this technology is the laparoscopic AR, where the images from the diagnostic studies are added to a 2D video camera image. Due to the deformation of the soft tissues from respiratory movements and pneumoperitoneum, the preoperative CT and MRI images cannot be superimposed in real time. Thus, to improve the orientation in the operating field topography, the surgeon tries to register stereoscopic video image into intraoperative ultrasound [15]. An interesting idea is an application of the cone beam CT (CBCT) mounted on a C arm for laparoscopic liver surgery, where the real-time graphics from CBCT are calibrated with the video image. An AR system Syngo iPilot® based on this idea was tested on the phantoms and was used in the laparoscopic resection of hepatocellular carcinoma (HCC) in the 7th segment of the liver of the 50-year-old patient. A maximal error of aim localization (guidance accuracy) in this system was reported to be only 2.49 mm [7]. One of the most advanced applications of AR in optical devices is the CAS-One (CAScination AG, Switzerland) with the da Vinci System robot (Intuitive Surgical Inc., USA) used in laparoscopic surgery. The 3D tumor and tools’ location are added to the endoscopic video image. AR is visible to an operating surgeon on the robot’s console. Detection of the tool position (camera and one of the robot’s arms) is facilitated with the placement of reflexive infrared-visible markers in an unchanging position of the tool. Then, the markers are followed with the stereoscopic optical camera (NDI Polaris® Vicra [16]) hanging over an operating field [17]. Unfortunately, CAScination AG seems to have concentrated on developing percutaneous tumor ablation navigation systems and otology and abandoned the project of AR laparoscopic surgery.

Monitors with AR, which allows fusion of the pre- and intra-operative data. Just like the devices from the first category described, a graphical image from a radiological study is superimposed on a real-time video image. Static, unchangeable point of view is also a limitation of this method, which requires the practitioner to move eye focus from the monitor to the operating field, as the visualization is not effective in situ. Additional calibration and registration in the OR are also inevitable [18]. Furnishing the system with the mobile video camera and reference plate eliminates the immobility of the viewpoint and the need for in-OR calibration. Visualization, however, still takes place outside the operating field [8]. The aforementioned CAS-One IR is a commercially available product with confirmed efficacy that fuses 3D video and CT images in real time on an external screen and can effectively guide needle tip for tumor ablation [19,20].

Window systems for AR features graphic information projected on a semi-translucent mirror placed between a surgeon and the field. The position of the screen, the patient, and the surgeon’s head are tracked to adjust the image projected on the mirror. The system requires registration and calibration inside the OR, and the window opened within the sight field may affect the surgeon’s performance [21]. An example of such a system (although mounted on the operator’s head) is a voice-guided Google Glass device, which can present any stored graphical information in the corner of the visual field or in a magnified form on a whiteboard outside the operating field.

Head-mounted devices (HMDs) allow in situ visualization with no need for a video camera, as they automatically and naturally adjust the viewpoint and spatial visualization without the engagement of the operator’s hands. Optical HMDs project graphic objects on 2 semi-translucent mirrors (separate for each eye). The positioning of the projection depends on an estimated distance from an operating field. Video HMDs create entire environments and separate the sight of an operator from the actual operating field. Head-mounted projector devices use a reflexive screen positioned close to the operating field. Graphical objects are then projected on the screen. HMDs require additional registration and are rarely accepted and used by clinicians [22,23]. However, the registration process in the HoloLens device is so effective that it can be medically applicable in the near future outside the Phillips Laboratory. One of the newest devices of this type with promising applicability is the Epson’s Moverio BT-300 goggles with an advanced Si-OLED display system [24]. These devices are currently used primarily in sports and recreation, to keep user hands free from the mobile phone while enabling calls and providing information.

The authors of this manuscript have developed another experimental AR system, MARVIS (Medical AR VISualizer) [25], which does not fit into any of the aforementioned categories. We used spatial recognition of the phantom surface and the operator’s head position in real time. The monoscopic projector hanging over an operating field displays the CT-acquired AR image of the target lesion hidden inside the phantom on its uneven surface, as seen from the operator’s point of view. Head movement changes the view angle and shape of the displayed image, enabling an operator to place the needle precisely in the hidden target (Figure 1) [25].

Figure 1

MARVIS (Medical AR VISualizer) augmented reality navigation system at work. (A) A marker on the operator’s head allowing video cameras to locate a head position. (B) Over-the-field projector displays anatomical features and the tumor view on the surface of the real, 3-dimensional model of the liver. (C) Targeting an aim hidden within the model with the needle connected to an electrical circuit. The closure of the circuit turns on the diode, which confirms the perfect placement of the needle.

AR in Transplantology

So far, there are no notions on the utilization of AR in organ transplantation. We are certain, however, that with the development of applications and popularization of AR outside the field of medicine, both mixed reality and AR will become important tools in organ transplantation. Although it has never been tried, AR could be used to promote organ donation. In 2016, the British National Health Service (NHS) launched a campaign encouraging blood donation. In short, when you put a sticker on your arm and scanned it to an iPhone or Android application you could actually see a needle and a tube put into your arm on the screen of your smartphone. If you held it long enough, an outdoor digital screen would display a bag of blood being filled and the condition of a real transfusion recipient improving. Finally, the outdoor screen displays thanks and your first name [26]. It is difficult to say how many additional blood donations this project was responsible for, but the campaign won an annual competition run by the Ocean Outdoor and the Campaign for the most creative outdoor advertisement idea. It definitely was a brilliant idea to attract young people to the issue of blood donation. Scenarios for similar AR applications could be developed for promotions of deceased and living donation, providing information for potential recipients, helping drug regimen adherence, and creating an educational environment for potential and actual recipients. AR could definitely improve deceased donation rates. At present, to obtain consent for organ donation, the transplant coordinators often demonstrate to the patient’s family the results of the brain angiogram. Using AR glasses, the coordinator could show to the family of the potential donor that the blood does not flow to the loved one’s brain. Such a proof of brain death would be a strong argument for obtaining the agreement for the organ’s donation. The AR has already been used to teach about the human blood flow more effectively [27,28]. Harvesting vascularized composite graft (VCA) is an extremely complicated procedure, requiring a lot of training and expertise. Preservation of the blood vessels to secure proper blood supply is of utmost importance. AR planning of microsurgery has been used to design surgical dissection for the superficial circumflex iliac artery perforator flap. 3D vascular anatomy can be retrieved from CT angiography and combined with the smartphone video camera image of the patient’s skin. Such a tool helped successful planning of the flap in 30 patients and reduced the time of the procedure by 20% [29]. As VCA retrieval lasts many hours, this can become a useful tool.

In addition, having scanned the donor and recipient at a certain stage of preparation for transplantation, a physician could align them in a unique reconstruction (registration) in a virtual or AR, and decide whether the donor’s liver, heart, or lung size match and would fit in the recipient’s body.

Virtual Surgical Training

The Cleveland Clinic, Ohio, USA in cooperation with Microsoft, construct a future health education facility with the core of their program the HoloLens generated AR. They provide students with a 21st-century holographic tool for learning gross anatomy, histology, embryology, radiology, and pathology. Thus, a teacher need not to be present in person in the class to assure effective learning [30]. The same technology or VR could be used to teach transplantation specialists to correctly assess an organ to be retrieved, take a biopsy sample using ultrasound or CT, interpret biopsy results, and perform laparoscopic or robotic organ retrieval and transplantation. Interactive teaching programs in AR or VR will eventually be a substitute for the courses requiring personal attendance offered currently by the transplantation societies. All educational UNOS Connect and ESOT programs like Hesperis, Academia, and LIDO (Living Donor Nephrectomy) hands-on courses could be organized in an AR environment with improved cost and time efficacy. One significant drawback will be the lack of personal contact, which builds team spirit, trust, and the desire to improve. Teaching transplantation surgery requires more experienced surgeon tutorship provided one on one to a pupil. The learning curve is long, and the teacher must be present in person during organ retrieval and transplantation. Expert surgeons are few and not likely to expand in number. With a video camera and AR goggles, an inexperienced surgeon could be instructed to identify vital anatomical structures, interpret the color and structure of the liver and pancreas, assess heart contractility, and the condition of the lungs. Steps of the retrieval and transplantation procedure could be followed and controlled from a distance, and even a small time-lag, which is an obstacle in the long-distance robotic surgery, would not be a limiting issue here. Similarly, a trainee could be instructed by an off-theatre mentor when performing transplantation. With a video camera mounted on the goggles used by the operator, an instructor could follow the whole procedure and indicate the vital structures or moves, displaying his remarks with AR directly on the operating field.

AR in the Assessment of Anatomical and Pathological Structures

As transplantation patients today have long life expectancy even while exposed to the immunosuppressive drugs, they suffer from the same diseases as the general age-matched population, with similar or increased morbidity. Non-Hodgkin lymphoma is 7 times, lung cancer 2 times, liver cancer 11 times, and kidney cancer 7 times more frequent in transplant recipients [31,32]. Kidney tumors occur both in native and transplanted kidneys and nephron-sparing surgery could be a reasonable therapeutic option. For example, Dutch researchers reported utilization of HoloLens visualized AR in planning surgery in 7 pediatric patients with Wilms tumor. They used commercially available software for image segmentation, 3D reconstruction, repair of minor artifacts, and creation of a proper application. The result was precise tumor visualization in relation to the vital anatomical structures including blood vessels and urinary collecting structures. Surgeons assessed AR holographic visualization as more informative and that they received a more detailed understanding of the kidney anatomy [33]. Such technology can be very useful in the planning of kidney retrieval from living donors, where a perfect assessment of kidney 3D anatomy, and especially vasculature, is key to the success of transplantation and donor safety. Approximately 9.4% of liver transplantation candidates in the USA need transplantation for hepatocellular carcinoma [7,34]. Almost 6% of these patients never make it to transplantation because of disease progression, hence, there is an urgent need for alternative or bridging therapies during enlistment procedures and while on the waiting list. Of the options available (chemotherapy, radiotherapy, chemo(radio)embolization, ablation, resection), surgical resection gives the best results in terms of long-term patient survival, with thermal ablation the second best. All these methods are also used to downstage tumor burden, and to transplant initially untransplantable patients, with acceptable results [35]. Even in low-risk patients, the tumor will recur in 15% to 20% of the transplanted livers [36], requiring intensive treatment. Besides, a good proportion of kidney transplant recipients have a history of hepatitis C virus or hepatitis B virus infection and are at a high risk of developing primary liver cancer. As recipients have routine Doppler ultrasound at least once a year, they are likely to be diagnosed in the early stage of the disease. Due to the position of the liver covered by the chest wall, its anatomical complexity and diversity, aiming at a tumor for biopsy to confirm the diagnosis, and to treat it with ablation, is relatively complicated. Resections, particularly made using the laparoscopic technique, require a lot of surgical expertise, and have a long learning curve. Application of AR with information on the anatomical location of the large vessels and the biliary tree could shorten the learning curve, improve patient safety and accessibility to treatment while enhancing effectiveness at the same time. Implementation of AR in liver surgery can help design the best access incision line and laparoscopic (or robotic) instrument ports placement, navigate the track and aim percutaneous biopsy, position the ablation electrodes or high-dose radiotherapy catheters, and orient tumor, large vessels, and bile ducts within the liver to preserve safe resection margin and avoid collision with the vital anatomical structures [4,17,37]. The University of Bonn has proposed a HoloLens and Moverio BT300 based system to navigate radiofrequency ablation needles into a liver tumor. The most difficult issue of target movement with respiration and liver deformation was dealt with in real time with the statistical model adjusting internal target position to the movement of the external markers from 4D-CT scans, aligned to the video image in a living animal. Using traditional CT-guided navigation, after 3 position adjustments, the needle was located 2.6 mm from the center of the lesion; AR navigation required only 1 adjustment to achieve similar accuracy [38]. Most researchers accept an error of 5 mm in locating the liver tumor. Of course, such real-time computations require fancy mathematical algorithms and a lot of computational power, and currently are inherently associated with a time lag between the actual and observed environment as the systems are available only to generate 4 frames of moving image per second [39].

Having the problem of respiratory movements of the target solved, the AR navigation could be applied to the biopsy of chest tumors in solid organ transplant recipients as well. In fact, the Navigation Technology team developed the image-guided video-assisted thoracoscopic surgery (iVATS) system, which uses an AR navigation system to remove early-stage lung tumors with minimally invasive surgery [40]. They performed a preliminary clinical trial in 23 patients with tumors 6 to 18 mm in diameter, and they achieved complete removal of the tumors with safe margins, and minimal post-operative complications. While the system was not very complicated, with the C-arm and fiducials to align CT scans to video image, the surgeon had to translate the 3D reconstructed CT and x-ray images with fiducials into real life [40]. As lung deflation is a major obstacle in following the lesion, a system to mark the landmarks in AR with a tooltip and continuous tissue tracking, is currently under development by Phillips [41]. Proper setup of future OR equipped with cyber-infrastructure allows displaying 3D lung dynamics from the deformable models on the surface of the patient’s body [42]. Emergency patients with pneumothorax, hydrothorax or, lung surgery patients, and transplantation patients could benefit from such AR-derived additional information. Lung cancer occurs more often in lung transplantation patients, and the PTLD (post-transplant lymphoproliferative disorder) derived lung tumors, affect almost 5% of patients, result in early mortality [43]. Breast cancer in transplantation patients has a similar occurrence to that of the general population but the results of treatment are much inferior [44]. Quality of Life Technology Labs designed an AR application that uses 3D mammography, breast tumor visualization, which renders its model on the patient’s body, using a head-mounted device and Android smartphone to facilitate screening and planning for breast surgery [45].

AR in the Treatment of Post-Transplant Pathologies

Colorectal cancers in solid organ transplant recipients occur only 10% more often than in the general population, yet the increase is mostly due to the proximal colon localization, which is less accessible and more difficult to diagnose early. Primary sclerosing cholangitis and cystic fibrosis patients are at significantly greater risk. Laparoscopic and robotic surgery is used more and more often for colon cancer resection. Although not quite AR, the TilePro multi-display was used for the presentation of 3D video image from an actual operating field, with the CT OsiriX-rendered 3D image of bowel anatomy next to it, in a pT4aN2b ascending colon cancer patient operated on with the DaVinci robot [46]. This technique prevented sight diversion from the operating field to the monitor, yet an operator still had to use spatial imagination to combine both images. The surgery was uneventful, and the surgeon assessed the 3D data presentation as very helpful. Similar technology was used to integrate real-time ultrasound data during robotic gynecological procedures [47].

Cardiovascular diseases are a leading cause of death in solid organ transplant recipients with a functioning graft [48]. Hence, a good proportion of these patients will require transcatheter procedures: angioplasty, stent placement, septal defect or foramen ovale closure, valve repair and other procedures, normally performed under x-ray fluoroscopy. Real-time 3De positioning was achieved in one study [49] using 2 fluoroscopic images taken from different angles registered to preoperative CT and displayed holographically with Microsoft HoloLens. Registration error was only 0.4 mm, and computational time lag was only 1.2 second, and granted better visualization to the interventionalist [49]. Polish cardiac surgeons used CT acquired images and AR to navigate during trans-catheter pacemaker placement in a patient with aberrant heart anatomy [50]. The same team reported successful revascularization of a chronically occluded coronary artery with AR visualization of the anatomy next to the operating field with the Google Glass [51].

End-stage renal disease patients transplanted for polycystic kidney disease are at increased risk of intracranial aneurysm rupture (5–9%) [49,52]. Application of AR with HoloLens device for the diagnostic studies of patients with internal carotid artery intracranial aneurysms allowed accurate blood flow and rupture risk assessment [53]. Careful planning of surgery with VR has provided excellent treatment results but needs a lot of expertise and mental transfer of the analyzed data into a real-life setting [54]. AR could simplify the process. Video images from the neurosurgical microscope can already be augmented with Doppler ultrasound and MRI data, to locate the lesion precisely [55].

AR microscope has been recently shown to improve histopathological diagnosis. Armed with a powerful artificial intelligence computer, the system displays areas of suspicion with the AR, through the eyepiece. Thanks to the high computational power, the AI is able to analyze a much higher number of fields than any pathologist in the same amount of time, and reports all the suspicious areas to an expert human for decision making [56]. Although computational algorithms are used today to help diagnose rejection in kidney, heart, and liver grafts while analyzing gene expression profile of a biopsy [57], the AR/AI-supported diagnosis platform requires less laboratory time, biostatistical analysis, and knowledge to process and understand the result.

Conclusions

Augmented reality (AR) is a state-of-the-art technology that is used primarily for entertainment and military applications. Although it slowly is being adopted in the medical field in radiology and surgery, it has never been tried in organ transplantation scenarios. However, it could be useful in the training of transplant surgeons, as well as in advertising organ donations, graft retrieval and allocation, microscopic diagnosis of rejection, treatment of complications, and post-transplant neoplasms. Availability of AR display tools like a Smartphone screen, and not too expensive head-mounted goggles, accessibility of software for automated image segmentation, 3D reconstruction, and algorithms allowing registration, make AR an attractive tool for surgery, including transplantation. A shortage of hospital IT specialists and abstaining of medical device manufacturers from investments in the development of AR technology remain the most significant obstacles to its broader application.