Co-axial Projective Imaging for Augmented Reality Telementoring in Skin Cancer Surgery.

Telemedicine has the potential to overcome the unequal distribution of medical resources worldwide. In this study, we report the second-generation co-axial projective imaging (CPI-2) system featured with orthotopic image projection for augmented reality surgical telementoring. The CPI-2 system can acquire surgical scene images from the local site, transmit them wirelessly to the remote site, and project the virtual annotations drawn by a remote expert with great accuracy to the surgical field. The performance characteristics of the CPI-2 system are quantitatively verified in benchtop experiments. The ex vivo study that compares the CPI-2 system and a monitor-based telementoring system shows that the CPI-2 system can reduce the focus shift and avoid subjective mapping of the instructions from a monitor to the real-world scene, thereby saving operation time and achieving precise teleguidance. The clinical feasibility of the CPI-2 system is validated in teleguided skin cancer surgery. Our ex vivo and in vivo experiment results imply the improved performance of surgical telementoring, and the clinical utility of deploying the CPI-2 system for surgical interventions in resource-limited settings. The CPI-2 system has the potential to reduce healthcare disparities in remote areas with limited resources.

Introduction

The achievement of universal health coverage is hindered by unequal distribution of medical resources across regions.[13] In low- and middle-income countries, lack of outstanding medical experts limits their provision of safe and affordable healthcare services.[23] Even in developed countries, health care disparities constrain the disadvantaged groups from obtaining sufficient resources.[12] Telemedicine has the potential to alleviate this inequality by bringing personalized healthcare and specialized surgical expertise closer to the patients in underdeveloped regions or desolated settings.[11] The prevalence of corona virus disease 2019 (COVID-19) further promotes the widespread implementation of telemedicine.[22] Telemedicine provides medical specialists with more opportunities for disease diagnosis, patient care, surgical guidance and collaboration, and surgical skills teaching without being restricted by distance.[14] Its clinical effectiveness has been validated in multiple medical conditions,[4] such as stroke,[20] heart failure,[16] and diabetes.[28]

Telementoring is a primary format of telemedicine where a clinical expert provides guidance to a less experienced clinician from a remote location. It can not only provide patients with more opportunities for surgical care but also enable surgeons to acquire innovative surgical techniques faster.[14] In the scenario of accident or natural disaster, a surgical telementoring system allows for medical experts to guide less experienced rescuers at the local site for medical interventions.[5,10] In the scenario of medical education, an experienced surgeon may use a surgical telementoring system to teach the complex surgical procedures interactively.[3] Surgical telementoring was first demonstrated in 1962 when Dr. Michael DeBakey performed an open heart surgery through video conferencing.[9] From then on, various telecommunication tools have been developed and tested for surgical telementoring in open and laparoscopic surgeries.[2]

Real-time video conferencing and augmented reality (AR) are commonly used methods for surgical telementoring. While many video conference systems allow instructors to draw annotations over the live video of the surgical site in order to provide both visual and verbal guidance, some systems only support oral guidance that places a great demand on instructors’ verbal ability.[3,6] In comparison, an AR system is able to superimpose virtual information and visual instructions on the real scene. Various wearable AR devices such as head mounted display (HMD) devices[26] and google glasses[30] have been explored for surgical navigation. Liu et al.[17] and Rojas-Munoz et al.[25] explored the use of Microsoft HoloLens for surgical telementoring and education in fasciotomy. However, widespread applications of these surgical telementoring systems are hindered by their limitations. First of all, many video-based telementoring systems use monitors to display surgical images and instructions. In order to follow the telementored instructions, the operators must look at the monitor, understand the instructors’ guidance, and look at the surgical site in order to map the instructions with the real-world scene.[1,27] The repetitive shift of attention between the surgical area and the monitor may not only waste the valuable surgical time but also cause possible errors due to the mismatch between the surgical area and the monitor display.[24] Second, wearable AR telementoring systems have limitations such as low comfortability for long-term use and poor co-registration of annotations with the 3D world. In order to overcome the above limitations and improve the clinical utility of teleguided surgery, we have previously designed a surgical telementoring system based on a co-axial projective imaging (CPI) module.[29] The CPI module comprises a camera and a projector co-axially arranged on the same optical path.

In this paper, we report the second-generation co-axial projective imaging (CPI-2) system for surgical telementoring, which further ameliorates the hardware and software design of the CPI module. The achievable resolution and accuracy of the CPI-2 system are characterized by benchtop experiments. The ex vivo study that compares the CPI-2 system and a monitor-based telementoring system shows that the CPI-2 system can reduce the focus shift and avoid subjective mapping of the instructions from a monitor to the real-world scene, thereby saving operation time and achieving precise teleguidance. Our clinical trial successfully demonstrates the clinical applicability of the CPI-2 system in skin cancer surgery with wide local excision and free flap transplantation. To the best of our knowledge, this is the first study that evaluates the utility and accuracy of a projective telementoring system for skin cancer surgery.

Materials and Methods

Surgical Telementoring Concept

Surgical telementoring involves real time interactions between an inexperienced operator at the local site and an experienced medical expert at the remote site. At the local site, a real time video streaming of the operation site is collected by a color camera in the CPI-2 system, color calibrated by our previous algorithm,[29] and transmitted to the remote site via a wireless network. At the remote site, the wirelessly transmitted video streaming of the surgical site is displayed in real time on a portable display device (e.g., a tablet) so that the surgical expert can make important medical decisions and provide interactive instructions. The user interface of the CPI-2 system supports the stylus function so that the surgical expert is able to create, delete, or modify annotations of any shape in the video streaming. The annotations will be transmitted wirelessly to the local site and projected orthotopically onto the surgical site in order to guide the local site for precise surgery. The schematic diagram of the CPI-2 surgical telementoring concept is shown in Fig. 1.

Figure 1

Schematic diagram of the surgical telementoring concept based on the CPI-2 system.

CPI-2 System

Hardware Design of the CPI-2 System

Figure 2a shows the setup of the proposed projective tele-mentoring system that consists of the CPI-2 module, a host computer and a medical trolley with a support arm. The CPI-2 module is installed on the support arm adjustable to fit the height of the operating table. The CPI-2 module consists of a MV-SUA230GC-T color CMOS camera (MindVision, Shenzhen, China), a CEL-5500 compact embeddable light engine (Digital light innovations, Texas, USA), a camera lens (Azure Photonics, Fujian, China), a relay lens (Edmund Optics China, Shenzhen, China), a 520 nm laser (OX Lasers China, Shenzhen, China), a beam block (Thorlabs, New Jersey, USA), as well as a beam splitter (Nano Macro Photonics, Shenzhen, China). The camera has a frame rate of about 60 frames per second, and a resolution of 1920 × 1200. A camera lens with a focal length of 50 mm is mounted on the camera in order to reach a field of view (FOV) from 100 mm × 72 mm to 132 mm × 95 mm within a working distance of 500 to 700 mm. Projection brightness at 500 mm working distance is about 9100 lux. The beam splitter with the transmission and reflectance ratio of 8:2 is mounted at an angle of 45° to the optical axis of the camera. This transmission to reflectance ratio is determined as a trade-off for optical intensity requirements between projection and imaging. The cost of the CPI-2 module is about $4000, of which the CEL-5500 compact embeddable light engine accounts for 75% of the cost. The cost of the host computer and the medical trolley is about $2000.

Figure 2

Hardware design of the CPI-2 system. (a) Trolley-mounted CPI-2 system; (b) Schematic diagram of the CPI-2 module.

The CPI-2 module is an elaborately designed optical system based on the principle of orthotopic projection where the camera and the projector are aligned at conjugate positions, as shown in Fig. 2b. In this design, the surgical scene images are reflected by the beam splitter and acquired by the CMOS camera, while the wirelessly transferred annotations are transmitted through the beam splitter and orthotopically projected onto the same surgical scene. This optical design enables high projection accuracy without the need for re-registration in a large range of working distances from 500 to 700 mm. More information on the CPI-2 system design and parts list can be found in Fig. S1 and Table S1 of the supplementary material.

Our previous CPI system, suffering from the limited brightness of a conventional projector, could not provide sufficient projection brightness for intraoperative guidance in the operating room with the surgical shadowless lamp turned on.[29] In the CPI-2 module, the conventional projector is replaced with the CEL-5500 compact embeddable light engine that can be connected to an external high-brightness light source. Green light from a 520 nm laser with a maximum power of 1 W is coupled into CEL-5500. The field of projection is modified to match the camera FOV, by replacing the original projection lens (37 mm focal length) with a 50 mm lens and a relay lens. The above efforts have greatly improved the projection brightness of our system to satisfy the use in an operating room where the shadowless lamp is on.

Software Design of the CPI-2 System

As shown in Fig. 3a, the user interface of the CPI-2 system includes three parts: a real-time video streaming display module, a control and annotation panel and a coaxial calibration panel. The designed software is installed on the local PC. The medical specialists in remote site can control the cursor of the local personal computer (PC) using a stylus or a mouse by a remote connectivity software during the surgical intervention. As the specialist draws annotations on the real time surgical field images, the drawn paths are generated and affine transformed based on the initial coaxial calibration parameters. Then the software in real time generates and transmits projection images with only white annotations on black background to the projector. The control and annotation panel also allows experts to easily adjust the width of annotations, delete annotations, and adjust the size of the image to match the real size of the local and the remote sites, as illustrated by the workflow chart in Fig. 3b. This software was developed in-house using Python (3.6), PyQt5 (5.13.0), and OpenCV (4.0.0). It could be found in https://github.com/FanZhang096/CPI-2-system.

Figure 3

Software design of the CPI-2 system. (a) User interface of the CPI-2 system; (b) The workflow chart.

Benchtop Experiments

In the proposed telementoring system, the projection accuracy of the CPI-2 module is critically important since it directly affects the annotation accuracy. We design a benchtop experiment to quantify the achievable accuracy of the CPI-2 system. Figure 4a shows the schematic diagram of this benchtop experiment. 16 corner points on a target board of 85 mm × 65 mm is captured and re-projected back by the CPI-2 system. The achievable accuracy of the CPI-2 module is obtained by calculating the average value of the Euclidean distances between the imaged and the projected corner points. It is also tested at different working distances (i.e., 500 to 700 mm) and working angles (i.e., 0°, ± 15°, and ± 30°), to simulate the patient’s curved body surface in real surgical scenarios. In addition, the resolution of the CPI-2 system is measured by capturing and re-projecting back the USAF 1951 resolution target at the working distance of 600 mm. The color accuracy is measured by the X-rite colorchecker passport.

Figure 4

Projection accuracy evaluation of CPI-2 system. (a) Schematic diagram of the experiment; (b) Projection errors at different working distances; (c) Projection errors at different working angles.

Low latency and high image quality during internet transmission are important for teleguided surgery. We also design a benchtop experiment to quantify the latency and degradation of image quality caused by internet transmission. First, the latency between the cities of Changchun and Hefei, which are about 1500 km away in a straight line, is tested. Second, the images of the 24-color card, the contrast card that contains different line pairs with line widths from 1 to 20 pixels, and the grayscale card that contains different grayscale values from 0 to 255 are transmitted from Hefei to Changchun. Finally, the degradation of image quality is determined by comparing the images transmitted from Hefei and the images received in Changchun. Degradation of image color accuracy and contrast are quantified using CIEDE2000[19] and Modulation Transfer Function (MTF-50).[21]

Ex Vivo Study

The CPI-2 system is compared with a monitor-based telementoring system in ex vivo study. An experienced medical expert in Ohio State University (OSU) remotely instructs six inexperienced trainees in University of Science and Technology of China (USTC) to perform simulated skin cancer surgery on the squid tissue. Three trainees use the CPI-2 system as an experimental group and others use a monitor-based system as a control group. Under the guidance of the expert, six trainees excise the simulated lesion and suture the squid tissue. The accuracy of the trainee's operation is determined by calculating the Intersection over Union (IOU)[15] of the region determined by the trainee and the region annotated by the expert. The operation time and the number of times a trainee shifts focus are also recorded. Operation time is defined as the time when the trainee determines the incision based on the expert's annotation. Focus shift is defined as a period of time when the trainee's attention is not in the operating field.

Clinical Trial

To verify the clinical feasibility of the CPI-2 system, the telementored skin cancer surgery is conducted. The clinical trial is approved by the ethics committee of the First Affiliated Hospital of USTC and adheres to the Declaration of Helsinki. A patient who needs surgery to remove skin cancer is recruited, and the written informed consent is obtained from the patient and the patient’s family. The telementored skin cancer surgery is performed between different districts of the first affiliated hospital of USTC. An experienced medical expert remotely guides a resident to complete the surgery through the CPI-2 system.

Results

Performance Characteristics

The results of the experiment that quantify the achievable accuracy of the CPI-2 system are shown in Fig. 4. The maximal projection error (i.e., the distance shift between the original and the projected images) is less than 1.1 mm for the entire FOV and the working distance ranging from 500 to 700 mm, as shown in Fig. 4b. The maximal projection error (i.e., the distance shift between the original and the projected images) is less than 1.8 mm for the entire FOV and the working angle ranging from − 30° to 30°, as shown in Fig. 4c.

The results of the experiment that quantify the resolution and color accuracy of the CPI-2 system are shown in Fig. 5. Figures 5a and 5b show the USAF 1951 resolution target imaged and reprojected by the CPI-2 system, respectively. Limits of resolving power are determined by the largest group of the element pairs where the spacing pattern is no longer discernible. The CPI-2 system is able to resolve the element 3 of group 1 with a corresponding linewidth of 0.198 mm in Fig. 5a as imaging and displaying on the host computer, and element 4 of group 0 with a corresponding linewidth of 0.354 mm in Fig. 5b as projecting on the target field. These two sets of elements are magnified and placed in the upper right corner of Figs. 5a and 5b, respectively, and the intensity profile of the two magnified regions along the dashed line are shown in Fig. 5c. The original image of the color card captured by the camera of the CPI-2 system is shown in Fig. 5d, and the image after color calibration is shown in Fig. 5e. Figure 5f shows the histogram of the ΔE (CIEDE2000) between the color values in Figs. 5d and 5e and the true values. Mean values of the ΔE (CIEDE2000) before and after color calibration are 9.91 and 1.33 respectively. ΔE < 2 after color calibration indicates that only experienced observer can notice the color difference.

Figure 5

Resolution and color accuracy evaluation of CPI-2 system. (a) Image of captured USAF 1951 resolution target; (b) Image of re-projected target; (c) The intensity variation of the region of interest (ROI) along the dashed line from image (a) and image (b); (d) Image of the captured X-rite colorchecker passport before color calibration; (e) Image (d) after color calibration; (f) Histogram of the ΔE (CIEDE2000) between the color values in image (d, e) and the true values. Mean values of the ΔE before and after color calibration are 9.91 and 1.33 respectively.

The results of the experiment that quantify the latency and degradation of image quality caused by internet transmission are shown in Fig. 6. Figures 6a–6c show original images of the 24-color card, the contrast card and the grayscale card. Figures 6d–6f show the images received in Changchun. At three different internet speeds, the maximum value of the latency is 70 ms, and the minimum value is 51 ms. The average values of ΔE (CIEDE2000) are less than 1 at three internet speeds, indicating that observer does not notice the color difference, as shown in Fig. 6g. The value of MTF is greater than 0.5 when the line width is larger than 1 pixel at three different network speeds as shown in Fig. 6h, which indicates that internet transmission has a significant effect on the contrast of target with a line width of 1 pixel. The grayscale values of grayscale target have not changed after internet transmission, as shown in Fig. 6i.

Figure 6

Evaluation of latency and image quality during internet transmission from Hefei to Changchun. (a–c) Original images of the 24-color card, the contrast card and the grayscale card; (d–f) The images received in Changchun; (g–i) Color, contrast, and grayscale difference at three different internet speeds. internet speed 1: 17 Mbps; internet speed 2: 37 Mbps; internet speed 3: 67 Mbps.

An experienced medical expert in OSU remotely instructs six inexperienced trainees in USTC to perform simulated skin cancer surgery, as shown in Fig. 7a. The CPI-2 system and a monitor-based system are shown in Fig. 7b. The real time video streaming of the operation site in USTC is wirelessly transmitted to OSU, in which the expert draws the surgical margin with a stylus. Two groups of trainees use blue ink to draw the incision margin according to the annotation directly projected on the tissue surface and the annotation displayed on the monitor respectively. Then they remove the simulated lesion and suture the squid tissue. The operating procedures are shown in Figs. 7c–7f. Table 1 shows the comparison results of two systems. The average IOU of the CPI-2 system and the monitor-based system are 0.94 and 0.81 respectively. The average operation time of the CPI-2 system and the monitor-based system are 44.7 seconds and 96.3 seconds respectively. The average number of times a trainee shifts focus of the CPI-2 system and the monitor-based system are 0 times and 8 times. The results of the ex vivo study show that compared with the monitor-based telementoring system, the CPI-2 system can reduce the focus shift and avoid subjective mapping of the instructions from a monitor to the real-world scene, thereby saving operation time and achieving precise teleguidance.

Figure 7

The ex vivo study that compares the CPI-2 system and a monitor-based system. (a) An experienced medical expert in OSU watches the transmitted video and draws the optimal trajectories for surgical excision; (b) A trainee removes the lesion according to the trajectory drawn by the expert; (c) The CPI-2 system projects the annotation drawn by the expert to the squid tissue; (d) The incision drawn by the trainee using blue ink; (e) The lesion removed by the trainee; (f) The squid tissue sutured by the trainee.

Table 1

IOU, operation time, and focus shift using the proposed system and a monitor-based system.

Parameter	CPI-2 system	Monitor-based system	p*
Accuracy (IOU)†	0.94 ± 0.01	0.81 ± 0.03	p < 0.05
Operating time†, s	44.7 ± 7.8	96.3 ± 11.0	p < 0.05
Focus shift†, times	0	8 ± 1	p < 0.05

*Values of p < 0.05 are statistically significant

†t-test

Telementored skin cancer surgery is performed between different districts of first affiliated hospital of USTC, as shown in Figs. 8a and 8b. During the wide-local excision process, the virtual annotation drawn by an experienced medical expert is transmitted to the CPI-2 system via wireless network and projected to the operating site. Then the resident surgeon draws the incision margin according to the annotation with methylene blue dye, and removes the lesion, as shown in Figs. 8c–8e. During the local flap reconstruction process, the above steps are also followed, as shown in Figs. 8f–8h. The removed lesion is sent to the pathology department for examination during the surgery. The pathological result confirms that it is basal cell carcinoma, and no cancer cells are seen at the margin, proving that it has been completely removed. A follow-up is conducted one month after the surgery, and the patient recovers well without any postoperative complications.

Figure 8

Telementored skin cancer surgery with wide-local excision and local flap reconstruction using CPI-2 system. (a) The CPI-2 system in operating room during the surgery; (b) An experienced expert is watching the images of the surgical field and drawing the optimal margin; (c) and (f) The annotations drawn by the expert are transmitted to the CPI-2 system and projected to the operating site for guiding wide-local excision and local flap reconstruction, respectively; (d) and (g) Following the projected annotations, the resident surgeon draws the margin with methylene blue dye; (e) The resident surgeon removes the lesion; (h) The resident surgeon performs local flap reconstruction.

Discussion

Many existing telementoring systems use a monitor to display the surgical field images and the remote annotations, leading to repetitive shift of the surgeon's attention between the surgical field and the monitor in order to follow the instructions. Other telementoring systems such as head-mounted AR systems suffer the low comfortability for long-term use and the poor co-registration between the annotations and the real world. In order to overcome these limitations, we propose a new design of telementoring that projects instructions directly to the surgical field with high precision. This design allows for a surgeon to focus on the surgical field without distraction. The proposed system uses a co-axial optical design to ensure accurate registration between the acquired image and the projected image with a positioning accuracy better than 1.8 mm in a large range of working distances from 500 to 700 mm. Our ex vivo study shows that the CPI-2 system can reduce the focus shift and avoid subjective mapping of the instructions from monitor onto the real-world scene, thereby saving the operation time and improving the teleguidance accuracy. In comparison with our previous CPI telementoring system,[29] the CPI-2 system adopts a new optical design less affected by the ambient shadowless lamp and uses a new software design to facilitate the tele-guidance process.

During ex vivo and in vivo experiments, the evaluations of experts and trainees are very positive. Since the CPI-2 system can intuitively display the annotations on the body surface, it is commented as interesting and practical. A total of 4 experts and 3 trainees experienced the CPI-2 system, and with our help, it only takes 5–10 min for them to become familiar with how to use the system. For experts, they find it easier to guide using a tablet equipped with a stylus. The biggest challenge is that the size of objects in the real world is inconsistent with the size displayed on the monitor. For this reason, our system has added the function of scaling images and annotations to match the actual standard size objects for calibration. Experts also suggest that software suitable for mobile phones can be further developed to suit a wider range of scenarios. For trainees, since the working distance of the CPI-2 system is 500–700 mm, there is enough space for operation and confirmation. In the future we will compare the results of people using the CPI-2 system with those using standard training methods in larger cohort studies.

Currently, the CPI-2 system still faces some limitations as a tool to benefit rural areas. First, it relies on wireless transmission of real-time high-quality data as other telementoring systems. The screen shot with a resolution of 1920 × 1080 and a frame rate of 60 Hz is transmitted to the remote site in real time, and the data size of a video with a duration of 1 second is 355.96 MB. The video is encoded before transmission by videoconferencing software to reduce data size while minimizing quality loss. When using the H.264 video coding standard, the required internet speed is 10 Mbps for a video with a resolution of 1920 × 1080.[18] The resolution and frame rate of the video will decrease accordingly when the internet speed decreases. At present, most of the world's land is covered by 4G signals, with an average internet speed of 17.6 Mbps.[8] In the future, a satellite communication device could be integrated into our system to allow communication via overhead satellites where cellular service and Wi-Fi are not available, and the introduction of 5G networks and the expansion of coverage could further reduce the latency and support higher image quality. Second, the specialist must redraw their annotations if the patient moves or the camera moves. Although some tracking algorithms can make the annotation follow the patient's movement, these algorithms usually rely on the texture and characteristics of the skin, resulting in poor robustness and applicability, especially when skin is excised or deformed. In the future, we plan to apply UV-sensitive photochromic nanocapsule tattoo ink[7] on the patient's skin, and excite it with UV light and galvano scanning mirrors to display specialist's annotation. This may be one of the effective ways to solve the tracking problem. Meanwhile, the low-cost galvano scanning mirrors can further reduce the cost of the system by replacing the higher-priced CEL-5500 compact embeddable light engine.

The CPI-2 system only provides images in visible light currently. In other surgical scenarios, such as sentinel lymph node exploration in breast cancer surgery, the invisible near-infrared wavelength range imaging is required. The CPI-2 system can integrate multimodal imaging devices, such as hyperspectral imaging for blood oxygen imaging; laser speckle imaging for blood perfusion imaging; fluorescence imaging for subcutaneous lymphatic structure and tumor localization. These modalities can be integrated into the image by modifying the CPI-2 system's light source, filter module and the camera. As the CPI-2 system is able to provide augmented information in situ, it could be also treated as an augmented reality (AR) device. Future work can be conducted in these directions.

To conclude, we present the second-generation co-axial projective imaging system for interactive telementoring during surgical interventions. This system is featured with real-time high brightness orthotopic image projection, higher accuracy, and fewer focus shift during the tele-guided surgery. Our benchtop experiments demonstrate that the CPI-2 system has adequate resolution and accuracy for medical image display and interactive telementoring. Our ex vivo and in vivo experiment results imply the improved performance of surgical telementoring, and the clinical utility of deploying the CPI-2 system for surgical interventions in resource-limited settings. The CPI-2 system has the potential to reduce healthcare disparities in remote areas with limited resources.

Below is the link to the electronic supplementary material.

Supplementary file1 (TIF 65779 kb)

Supplementary file2 (DOCX 15 kb)