Coaxial projective imaging system for surgical navigation and telementoring.

A coaxial projective imaging (CPI) module acquires surgical scene images from the local site of surgery, transfers them wirelessly to the remote site, and projects instructive annotations to the surgical field. At the remote site, the surgical scene images are displayed, and the instructive annotations from a surgical specialist are wirelessly transferred back to the local site in order to guide the surgical intervention by a less experienced surgeon. The CPI module achieves seamless imaging of the surgical field and accurate projection of the instructive annotations, by a coaxial optical path design that couples the imaging arm with the projection arm and by a color correction algorithm that recovers the true color of the surgical scene. Our benchtop study of tele-guided intervention verifies that the proposed system has a positional accuracy of better than 1 mm at a working distance ranging from 300 to 500 mm. Our in vivo study of cricothyrotomy in a rabbit model proves the concept of tele-mentored surgical navigation. This is the first report of tele-guided surgery based on CPI. The proposed technique can be potentially used for surgical training and for telementored surgery in resource-limited settings.

Introduction

Despite recent advances in surgical science, further dissemination of modern surgical techniques to impact human health and welfare is hindered by healthcare disparities, especially among socially disadvantaged populations who may have only limited resources. The emerging field of telemedicine has the potential to overcome such disparities by bringing personalized healthcare and surgical expertise closer to patients in underdeveloped regions or isolated settings. If emergency care is needed in such a setting, a telemedicine tool may help a medical specialist guide surgical intervention when the personnel on site have little or no relevant experience., In addition, a telemedicine tool may help in an educational setting to allow an experienced surgeon to teach complicated surgical procedures in an interactive way that may be superior to conventional educational tools such as video conferencing., By using a telemedicine tool, a less experienced surgeon at the site of surgery is able to share the same sight of view with a surgical specialist at the remote site, enabling interactive knowledge transfer, seamless decision making, and instructed intervention without a distance barrier.,

Surgical telementoring uses telecommunication techniques to provide interactive guidance and technical assistance from distance to operating surgeons., Since the first clinical demonstration of video conference-guided open-heart surgery in 1962, various telecommunication tools and augmented reality systems have been tested clinically for surgical telementoring. Shenai et al. successfully demonstrated a virtual interactive presence and augmented reality (VIPAR) system in a cadaveric carotid endarterectomy and a craniotomy. Andersen et al. designed and implemented STAR, a system for telementoring with augmented reality, which used a virtual transparent display to convey precise locations in the operating field to a trainee surgeon. Furthermore, various wearable devices such as HMDs and goggles have been explored for surgical navigation and telementoring., However, previous surgical navigation and telementoring systems have various limitations that hinder their broad clinical adoption. In particular, many telementoring systems use a monitor to display the surgical images and instructions. During a tele-guided surgical intervention, the operating surgeon has to switch his/her sight of view between the surgical field and the monitor in order to follow the instructions. This mode of operation may not only waste precious surgical time but also induce possible errors due to the mismatch between the surgical field and the monitor display. Wearable navigation systems have other limitations, such as low resolution, poor coregistration, limited field of view (FOV), blockage of natural vision, short battery life, and overheating. As such, current telementoring techniques are still suboptimal and cannot be broadly deployed for tele-guided surgical applications.

To overcome the aforementioned limitations and facilitate broader clinical deployment of tele-guided surgery, we propose a projective telementoring and surgical navigation system. As shown in Fig. 1, the system consists of a local site and a remote site connected through a wireless network. At the local site, the surgical scene is acquired by a coaxial projective imaging module (CPI-1), processed by a computer, and transmitted through the wireless network to the remote site. At the remote site, the wirelessly transmitted images of the surgical scene are displayed in real-time on a portable display device (e.g., a computer, a tablet, or a cellphone) so that a surgical specialist is able to make critical medical decisions and provide interactive instructions. The annotated instructions by the specialist will be transmitted back and projected onto the surgical site in order to guide the less experienced healthcare personnel about appropriate interventions with high accuracy. To the best of our knowledge, this is the first report on a projective telementoring and surgical navigation system and the first demonstration of its clinical utility in both benchtop and in vivo models.

Fig. 1

Schematic diagram of the proposed projective telementoring and surgical navigation system.

Material and Methods

Materials

The key component of the proposed projective telementoring and surgical navigation system is a CPI-1 coaxial projective imaging module, which is capable of acquiring the surgical scene images and displaying the annotations simultaneously at the site of surgery. The CPI-1 module consists of a MV-GED130C-T color charge-coupled diode (CCD) camera (MindVision, Shenzhen, China), a PRO4500VIS700 projector (Wintech, Shanghai, China), a 532-nm NF-25C05-40-532 notch filter (OptoSigma Corporation, Japan), and a 530 nm band-pass filter (Zhaojiu Photoelectric, Shanghai, China), as well as a beam splitter (Nano Macro Photonics, Shenzhen, China), as shown in Fig. 2(a). In our previous design of the projective imager, the CCD camera and the projector are placed side by side, inducing the mismatched fields of view that need to be calibrated at different working distances. To eliminate such coregistration error between the acquired and the projected images, we propose a coaxial optical path design as shown in Fig. 2(b). In this design, the surgical scene image is reflected by the beam splitter and acquired by the CCD camera, and the wirelessly transferred annotations are transmitted through the beam splitter and projected onto the same surgical scene with accurate coregistration regardless of any changes of the working distance. To avoid interference of the projected annotations with the surgical scene images acquired by the CCD camera, a 530-nm band-pass filter and a 532-nm notch filter are placed on the projector arm and the camera arm of the coaxial optical path, respectively. With this design, the projected annotations at the surgical site will not affect the surgical scene images acquired by the CCD camera. The CCD camera has a chip sensor with the size of 1/3″, a dynamic range of 12 bits, a frame rate of about 30 frames per second, and a resolution of . A prime lens with a focal length of 12 mm is mounted on the camera in order to reach an FOV of for surgical navigation in a working distance of 300 mm. The 532-nm notch filter in front of the CCD camera is fixed on a flip mount that is freely accessible for a 0 deg to 90 deg range, enabling us to switch the notch filter on or off for different tasks of image acquisition or color correction. The projector perpendicular to the camera projects instructive annotations or other images to the surgical field at a contrast ratio of 1000:1 and a resolution of . The 530-nm band-pass filter is mounted in front of the projector. The beam splitter with the transmission and reflectance ratio of 80:20 is mounted at an angle of 45 deg to the optical axis of the camera. This transmission to reflectance ratio is determined as a trade-off for optical intensity requirements between projection and imaging. The position and the orientation of the projector are aligned carefully for coincident projection and imaging. A TeamViewer software package (TeamViewer GmbH, Göppingen, Germany) that is compliant with the Health Insurance Portability and Accountability Act is used for wireless data transmission between the local and the remote sites. At the local site, a computer is used for image acquisition, projection, and processing tasks. At the remote site, a portable display device is used for surgical field display and instructive annotation.

Fig. 2

Telementoring and surgical navigation system at the local site. (a) System setup. (b) Schematic diagram of the CPI-1 module.

Color Correction Algorithm

To facilitate effective decision-making and instructive annotation by the surgical specialist at the remote site, the true color display of the surgical scene is preferred. However, due to the use of the 532-nm notch filter on the camera arm of the CPI-1 module, the surgical field images acquired by the CCD camera lack the color component at 532 nm. In other words, the green channel is much weaker than the other two. Therefore, color correction is required in order to recover the true color of the surgical scene.

Our color correction algorithm is applied to an RGB color image using a transformation matrix. The conventional method of color correction is to scale red, green, and blue channels. Since this method only considers the effects of a single channel, it induces a certain error in color recovery of chromatic images. To overcome this limitation, we modify the conventional color correction algorithm by inducing multiple color components as where is a matrix of transformation parameters; are the normalized pixel values in red, green, and blue channels of the image captured without a 532-nm notch filter; are the normalized pixel values in red, green, and blue channels of the image captured with a 532-nm notch filter.

The green channel is compensated by introducing a color correction card that has 104 different colors and four circular markers on the corners. First, we acquire one color card image that contains four circular markers in its FOV with the notch filter and another image without the filter. The two images are automatically coregistered according to the four markers. Second, the color blocks at the same position in two images are automatically cropped. Finally, the transformation matrix is obtained by Eq. (1), where are the mean color values of each color block cropped from the image captured without the notch filter, and are the mean color values of each color block cropped from the image with the notch filter. Python is used for programming the above image processing tasks.

The correction matrix obtained from the above color correction process is used to recover colors for the video streams acquired by the CCD camera with the notch filter. The computer at the local site is responsible for acquiring the color images of the surgical scene, correcting the weak green channel, and transferring the corrected images to the remote site. In the meanwhile, the computer at the remote site receives and displays the true-color surgical scene images after color correction, where the experienced surgeons can make surgical annotations. The annotations are transferred back to the local site through TeamViewer software and projected to the surgical site for guided surgical intervention

Animal Experiment

Telementored cricothyrotomy in a rabbit model is used as a proof of concept to demonstrate the clinical utility of tele-guided surgical interventions. Cricothyrotomy is a procedure aimed at creating an airway by incision for a certain life-threatening situation with low complications. Telementored tracheal intubation may add another dimension to airway management by avoiding multiple attempts at intubation and preventing iatrogenic injuries in the emergency setting. In addition, the proposed model of telementored cricothyrotomy also provides a valuable teaching tool for medical students to learn clinical anatomy during surgery.

A rabbit of 6- to 7-week-old with a weight of about 200 g is used for the animal validation tests. The test protocol is approved by the Institutional Animal Care and Use Committee of University of Science and Technology of China. The animal is anesthetized by intraperitoneal injection of urethane (20% concentration). After anesthesia, the rabbit is fixed on the operating table, with its four limbs stretched out and secured by the medical tapes. During our experiments, the rabbit is monitored every 5 min in case our experiment may produce extra pain. The CPI-1 module is installed 300 mm above the operating table for image acquisition and annotation projection at the surgical site. At the remote site, an experienced surgical specialist watches the wirelessly transferred surgical scene image and makes the instructive annotations for remote guidance of cricothyrotomy. The annotations are transferred back to the local surgical site and projected onto the surgical field to guide a less experienced surgeon for appropriate incision.

Results

Accuracy of the Color Correction Algorithm

The accuracy of the color correction algorithm is evaluated by calculating the Euclidean distance of the color differences in our projective telementoring system. The Euclidean distance is the distance between two colors within a color space such as RGB, Luv, and Lab. The International Commission on Illumination (CIE) chooses the distance metric to denote the color difference in the space and the most recent evaluation standard is , which resolves the perceptual uniformity issue.,

We use the evaluation standard to test the accuracy of our color correction algorithm in both a color card and a hand. Figure 3 shows the color correction results of the color card. Figures 3(a)–3(c) show the images of the color card captured with the notch filter, without the notch filter, and with the notch filter but after color correction, respectively. The color difference between Figs. 3(b) and 3(c) is calculated by the CIDE2000 color difference formulas and plotted in Fig. 3(d) as a histogram. According to the evaluation criteria of CIDE2000, a level between 1 and 2 implies that only experienced observer can notice the color difference. The averaged of the color card is 1.58, indicating that the color card image shows no significant difference from the original image after color correction.

Fig. 3

The color correction result of the color card. (a) Image with the 532-nm notch filter. (b) Image without the 532-nm notch filter. (c) Image with the notch filter but after color correction. (d) Histogram of the (CIEDE2000) between the image (b) and the image (c) after color correction. Mean value of the is 1.58.

Similarly, we also test the color correction outcome for the hand. Figures 4(a)–4(c) show the images of the hand captured with the notch filter, without the notch filter, and with the notch filter but after color correction, respectively. Figure 4(d) shows the histogram of the color difference between Figs. 4(b) and 4(c). The averaged of the hand is 1.37, indicating that the hand image shows no significant difference from the original image after color correction.

Fig. 4

The color correction result of a hand. (a) Image with the 532-nm notch filter. (b) Image without the 532-nm notch filter. (c) Image with the notch filter but after color correction. (d) Histogram of the (CIEDE2000) between the image (b) and the image (c) after color correction. Mean value of the is 1.37.

Color Correction for the Remote Expert

The color of the image seen by a remote expert through a PC or a tablet is affected by the following factors: (1) color correction accuracy for the images acquired at the local surgical site, (2) compression and decompression of the image data, and (3) color display accuracy by the remote expert through the PC or tablet. Our color correction algorithm described above has calibrated the first factor to ensure true color display at the local surgical scene. To calibrate the second factor, we transfer the image of a 24-color card between two cities in China (Hefei and Changchun) through TeamViewer’s quality priority mode at different bandwidths (i.e., 1.15, 9.98, and 21.84 Mbps). The average values of (CIEDE2000) are all less than 1 at three network bandwidths, indicating no observable difference at different network conditions. Further study shows that compression and decompression of image data by a commonly used telecommunication software package such as TeamViewer will not induce significant variations in image color as long as the network bandwidth between two places is greater than 1.15 Mbps. In terms of the third factor, the remote display should be color calibrated using an International Color Consortium (ICC) color correction file or a monitor color correction instrument. We choose to present the color calibrated terminal screen to the clinicians in order to ensure that no color distortion is added on the remote display. The color display of the terminal screen (AOC LV253WQP) is first calibrated using a Spyder 4Express display color correction instrument and a DisplayCAL color correction software package. After calibration, the color evaluation is performed using a DisplayCAL color correction software package. The averaged (CIEDE2000) level between the standard and the calibrated 51 colors is 0.46, indicating negligible color difference at the remote terminal site.

Accuracy of the Telementoring and Surgical Navigation System

The achievable accuracy of the proposed telementoring system is verified in two benchtop experiments as shown in Fig. 5. In the first experiment, a cardboard of with a matrix of white circles printed on the black background [Fig. 5(a)] is imaged and projected back onto the target cardboard. Both the original and the projected images of the white circles are acquired subsequently by the same camera. We have tested the projection accuracy of the target cardboard at different working distances (i.e., 300 to 500 mm) and at different angles (i.e., 0 deg, 30 deg, and 60 deg) in order to simulate the biological tissue of different locations and curvatures. According to Fig. 5(b), the maximal projection error (i.e., the distance shift between the original and the projected images) is less than 0.7 mm for the entire FOV and for the working distance ranging from 300 to 500 mm. According to Fig. 5(c), the maximal projection error (i.e., the distance shift between the original and the projected images) is less than 2 mm for the entire FOV and for the working angle ranging from 0 deg to 60 deg.

Fig. 5

The evaluation results for projection accuracy. (a) The target cardboard with a matrix of white circles. (b) Results of the central distance shift between the actual circle and the projected circle at various working distances (i.e., 300, 400, and 500 mm). (c) Results of the central distance shift between the actual circle and the projected circle at various working angles (i.e., 0 deg, 30 deg, and 60 deg). (d) Results of four rectangular markers placed by volunteers to record four corner positions of the projected rectangle. (e) Maximal positional offsets for the rectangular markers placed by the volunteers.

In the second experiment, rectangles of different sizes are generated by the computer at the remote site, transferred back, and projected by the CPI-1 module within a surgical field of . Volunteers are asked to place four rectangular markers at four corners of the projected image by aligning two edges of the marker with the corresponding edges of the projected rectangle, as shown in Fig. 5(d). The image of the projected rectangle and the image of four aligned markers are acquired by the CPI-1 module, and the coordinates of four corner points are detected by the Harries–Stephens algorithm. Positional deviations between the projected image and the marker image are calculated at each of the four corners and plotted in Fig. 5(e). According to Fig. 5(e), the image-guided operation by the proposed telementoring and surgical navigation system may induce a positional error of less than 1 mm in both and directions.

In Vivo Demonstration of Telementored Surgery

Clinical utility of the proposed telementoring system is demonstrated in vivo using cricothyrotomy in a rabbit model as a proof of concept. Figures 6(a) and 6(d) are pictures captured at the remote site and the local surgical site, respectively. Figures 6(b), 6(e), and 6(f) demonstrate the guidance for the location of the initial incision. Figure 6(b) shows the main annotation of the surgical incision drawn by the surgical specialist on the remote desktop. Figure 6(e) refers to the surgical instructive annotation superimposed into the actual surgical field. Figure 6(f) is the result of the incision. Figures 6(c), 6(g), and 6(h) show the simulated surgical procedure of tracheal intubation during cricothyrotomy. Figure 6(c) refers to the incision annotation of the tracheal intubation position drawn by a surgical specialist on the remote screen. In the meanwhile, this incision annotation is superimposed into the surgical field as shown in Fig. 6(g) to provide localized guidance for cricothyrotomy. Figure 6(h) shows the experimental result of telementored cricothyrotomy. In this demonstration, an operator without any knowledge of clinical anatomy is able to perform the cricothyrotomy procedure correctly following the remote guidance of an experienced surgical specialist.

Fig. 6

Cricothyrotomy surgery on the rabbit. (a) Overall view of the remote guidance of the surgical scene. (d) Overall view of the surgical field. The red arrow in (b) refers to the surgical incision on the remote screen, in (e) refers to the instructive annotation superimposed into the surgical field, and in (f) refers to the result of the surgical incision, respectively. The red arrow in (c) refers to the position of the endotracheal intubation on the remote screen, in (g) refers to the instructive annotation superimposed into the surgical field, and in (h) refers to the result of the cricothyrotomy, respectively.

In addition to tele-guided surgery, the projective imaging system can also be used as a telementoring tool for medical education. As shown in Fig. 7(a), the experienced surgical specialist remotely makes the surgical annotations and draws the primary anatomical structures on the touch screen that displays the real-time surgical scene. Figures 7(b) and 7(c) show the actual projection window (i.e., the surgical field) before and after superimposition of the instructive annotations remotely provided by the surgical specialist. The seamless coregistration between the annotations and the surgical scene enables interactive teaching of clinical anatomy and surgical skills without any distance barrier.

Fig. 7

Interactive instruction of clinical anatomy by telementored surgery in a rabbit model. (a) Touch screen display of the surgical scene and the surgical specialist’s annotations at the remote site. (b) The actual surgical field before projection of annotations. (c) The actual surgical field superimposed with instructive annotations provided remotely by the surgical specialist. Red arrows refer to the instructive annotations.

Discussion

Many existing telementoring systems in open surgery display the surgical scene images in a separate monitor, and an operating surgeon has to switch his/her sight of view between the surgical field and the monitor in order to follow the annotated instructions. Other telementoring systems, such as a DaVinci Surgical system, provide three-dimensional (3-D) visualization through a stereo viewer that allows for direct annotation. However, the cost for purchasing and maintaining a Da Vinci Surgical system is very high, making it unaffordable in regions of limited resources. Besides, telementoring based on a stereo viewer is more suitable in a minimally invasive surgery instead of an open surgery. For the first time, we propose a telementoring system specially used in open surgery that overcomes the current surgical limitations by projecting the annotated instructions directly to the surgical field with high accuracy. Such a telementoring design enables an operating surgeon to focus on the field of surgery without distraction. Compared with our previous projective imager that required frequent calibration at different working distances, the proposed system chooses a coaxial optical path that ensures accurate coregistration between the acquired and the projected images regardless of any changes of the working distance. In comparison with similar surgical navigation devices that overlay fluorescence images in the field of surgery,, we acquire true color images of the surgical scene and project the annotations at a visible wavelength without interference by using the specific filters. A color correction algorithm is developed to recover the original color of the surgical scene for display at the remote site. Technical feasibility of tele-guided intervention is verified by benchtop experiments, with a positional accuracy better than 1 mm within the working distance of 300 to 500 mm. Clinical utility of tele-guided cricothyrotomy is demonstrated in a rabbit model in vivo.

While the performance of the projection would be influenced by the depth variation of the 3-D object, there are two parameters we want to discuss about. One is “defocus,” and the other is “accuracy.” The projection will become a little bit blurry when part of the object is out of focus to the camera and the projector. To compensate this defocus effect, we have adjusted the camera lens to a larger (smaller aperture) to extend its depth of field (DOF). Furthermore, to reduce the defocus effect from the projector side, a laser scanning-based projector will be used in the future, to extend the projector’s DOF to “infinity.” In addition, the scattering effect of the biological tissue will also reduce the visualized difference between the in-focus and out-of-focus projection patterns. At the current configuration, the defocus effect has not caused significant inconvenience when conducting experiments on the rabbit. For the system accuracy, the depth variation will not induce additional projection error since the current design of the coaxial optical path ensures coincident optical axes for the camera and the projector, leading to the coincidence of the projected image with the real subject at different working distances.

Further engineering optimization and in vivo validation works are required before translation and dissemination of the proposed telementoring technique in a clinical setting. First, the current system uses a public accessible wireless network and a free software package (i.e., TeamViewer) for data transfer between the local site and the remote site. This design not only makes the speed of the telementoring system vulnerable to the network environment but also induces cyber-security concerns. In our preliminary study, the latency of network transmission between Columbus and Hefei, China, was estimated using the TeamViewer software package. The latency was found to be around 300 ms at a download rate of 32.08 Mbps and an upload rate of 16.83 Mbps in Hefei, and a download rate of 20.31 Mbps and an upload rate of 7.07 Mbps in Columbus. The network transmission latency in telemedicine can be further reduced by using a satellite, wired, or 5G network. For clinical deployment of telementoring, a secure protocol for wireless communication and a specially designed software package are required. Second, to facilitate dissemination of the proposed telementoring system in a clinical setting, many operating parameters, such as the working distance between the projector and the field of surgery, the FOV, and the projector intensity, need to be optimized. At the current stage of preliminary study, the contrast ratio and the brightness of the projective telementoring system is insufficient for use in an operating room with the surgical lights on. In the future, many methods, such as modulation of the surgical light, increase of the projector intensity, and laser-based projection, will be explored for enhanced clinical utility in an operating room. Third, the available display and input platforms in telementoring depends on software support for telecommunication. Among the PC, tablet, and cell phone platforms we have tested, a tablet equipped with a stylus provided a convenient and natural way for tele-annotation. In the future, we will systemically compare the annotation consistency and accuracy characteristics across different input and display platforms in order to optimize the telementoring performance for the best user experience. We will also explore other telecommunication tools and custom program the telementoring functions for the optimal annotation performance. Finally, telementored cricothyrotomy is used as a proof of concept to demonstrate the clinical utility of the proposed system. Since cricothyrotomy may not be the most suitable intervention for tele-guidance, and since the animal procedure does not resemble that in a clinical setting, appropriate selection of the surgical scenario and extensive clinical validation are required before clinical dissemination of the proposed telementoring technique. Nevertheless, the proposed projective imaging and telementoring system may be used for tele-guidance of many surgical interventions.

Conclusion

We report a surgical navigation system that integrates CPI and telecommunication for interactive telementoring during surgical interventions. Technical feasibility of tele-guided interventions is verified by benchtop experiments, and clinical utility of the proposed system is demonstrated by tele-guided cricothyrotomy in a rabbit model in vivo. Although further clinical validation is needed for dissemination of the proposed telementoring system in a clinical setting, our research has the potential to overcome healthcare disparity by bringing personalized healthcare and surgical expertise closer to patients.