Real-time Near-infrared Virtual Intraoperative Surgical Photoacoustic Microscopy.

We developed a near infrared (NIR) virtual intraoperative surgical photoacoustic microscopy (NIR-VISPAM) system that combines a conventional surgical microscope and an NIR light photoacoustic microscopy (PAM) system. NIR-VISPAM can simultaneously visualize PA B-scan images at a maximum display rate of 45 Hz and display enlarged microscopic images on a surgeon's view plane through the ocular lenses of the surgical microscope as augmented reality. The use of the invisible NIR light eliminated the disturbance to the surgeon's vision caused by the visible PAM excitation laser in a previous report. Further, the maximum permissible laser pulse energy at this wavelength is approximately 5 times more than that at the visible spectral range. The use of a needle-type ultrasound transducer without any water bath for acoustic coupling can enhance convenience in an intraoperative environment. We successfully guided needle and injected carbon particles in biological tissues ex vivo and in melanoma-bearing mice in vivo.

Introduction

During microsurgeries, visualization of sub-surface information is crucial to improve the accuracy of incisions and suturing, and to prevent unintentional accidents such as copious bleeding and tissue damage. Thus, since the early 20th century, intraoperative surgical microscopes have been regarded as essential devices for microsurgeries in ophthalmology, orthopedic surgery, neurosurgery, plastic surgery, and so forth [1-3]. Although the use of an optical microscope increases the surgical accuracy and efficacy during the microsurgery, it only provides magnified surface images within the region of interest; it cannot provide sub-surface information. To overcome this limitation, intraoperative imaging methods such as X-ray imaging, computed tomography (CT), ultrasound (US) imaging, and magnetic resonance imaging (MRI) have been adapted for use in surgical environments before, during, and after surgery [4-7]. However, these intraoperative imaging methods cannot maximize the surgical capabilities due to either ionizing radiation, low spatial resolution, low sensitivity, inconvenience, bulkiness or slow image acquisition.

Photoacoustic microscopy (PAM) is an emerging medical imaging modality based on optical excitation and US detection via light induced thermoelastic expansion [8,9]. PAM is capable of supplying sub-surface anatomical as well as functional, metabolic, molecular, and genetic information in real time [10]. Thus, this imaging method has been used in both clinical and preclinical research in several medical fields [11-21].

A virtual intraoperative surgical photoacoustic microscope (VISPAM) has been developed and used to guide needle insertion into live animals [22], but this system has several disadvantages. It uses a green (i.e., wavelength λ = 532 nm) laser beam as a PA excitation source, and this visible light significantly disturbed the surgeons’ vision during in vivo experiments. Further, the VISPAM B-scan image was displayed at 2 Hz, which was not fast enough for real-time imaging. In addition, VISPAM entails use of a water bath for acoustic coupling, and this device limits the maximum capability of the system in surgical conditions.

In this article, we describe a real-time near-infrared virtual intraoperative surgical photoacoustic microscopy (NIR-VISPAM) system that combines commercial surgical microscopy and PAM with an invisible NIR laser source (i.e., λ = 1064 nm). By sharing the same optical path, the NIR-PAM system was easily adapted to the conventional optical microscope; the NIR laser light is invisible, so it did not annoy the operators during surgery. Other benefits include a deeper penetration of NIR light than green light into tissue, and a higher laser safety limit (i.e., 100 mJ/cm2 at λ = 1064 nm vs. 20 mJ/cm2 at λ = 532 nm). Further, the conventional microscopic and PA B-scan images were displayed concurrently on the microscopic view plane using augmented reality. The PA B-scan image display rate reached maximally up to 45 Hz, so the real-time imaging capability was achieved. Moreover, a custom-made needle US transducer eliminates the need to use a water bath, which is closer to real clinical practice. The axial and lateral resolutions were 61 ± 1.4 and 36 ± 0.9 μm, respectively. We used the system to guide needle insertion and to monitor injection of carbon particles into chicken tissue ex vivo and into melanoma-bearing mice in vivo.

Material and methods

The NIR-VISPAM system (Fig. 1a, b) consisted of an NIR pulsed laser source (Teem photonics, SNP-20F-100) as a main PA excitation source; a per-pulsed laser energy of 4 μJ, a repetition rate of 20 kHz, a pulse width of 0.7 ns, and λ = 1064 nm. Initially, 10% of the laser light was deflected by a beam splitter (Thorlabs, CM1-BP108) and directed into a photodetector (Thorlabs, PDA36A-EC) to trigger a galvo-scanning mirror and a data acquisition (DAQ) system. The remaining 90% of the light was delivered to the NIR-VISPAM system. Then the NIR-VISPAM system was implemented by modifying a commercial surgical microscope (Carl Zeiss, OPMI). The NIR-VISPAM system consisted of three main divisions: (i) a customized PAM scanning [D1], (ii) a beam-projecting [D2], and (iii) a beam-splitting [D3].

Fig. 1

(a) Schematic of the near-infrared virtual intraoperative surgical photoacoustic microscopy (NIR-VISPAM) system. (b) Photograph of the NIR-VISPAM system. COM, computer; PD, photodiode; BS, beam splitter; NF, neutral density filter; AMP, amplifier; BP, beam projector; M, mirror; G, galvo-scanner; OL, object lens; NUT, needle type ultrasonic transducer; and PCI DTZ, PCI digitizer.

The PAM scanning [D1], used three devices: (1) a two-dimensional galvanometer (Thorlabs, GVS002) to scan the laser beam in the X-Y plane; (2) an objective lens (Thorlabs, AC254-075-B; diameter: 25.4 mm, focal length: 75 mm, NA: 0.17); and (3) a dichromatic mirror (Edmund optics, NT55-233) to reflect the NIR PA excitation light to the sample and to transmit the native visible light from the sample surface to the surgical microscope. Pulsed NIR irradiation stimulated emission of PA waves, which were detected by a homemade needle-type transducer with a length of 48.5 mm, a diameter of 1 mm, and a central frequency of 41 MHz (University of Southern California). Instead of a water tray, the needle transducer was directly coupled to the targets by ultrasound gel. The acquired PA signals were amplified by two successive amplifiers (Mini-Circuits, ZFL-500LN + ), then digitized by the DAQ board (NI instrument, PCI-5124). One-dimensional optical scanning along the X-axis acquired data for one depth-resolved PA B-mode image. The typical pixel numbers along X and Z axes in one PA B-mode image were 200 and 1800, respectively. The Hilbert transform was applied along each PA A-line. The maximum image display rate of one reconstructed PA B-mode image was 45 Hz. To increase the signal to noise ratio (SNR), two and three PA B-mode images were averaged for in vitro and in vivo experiments, respectively.

Beam projection [D2] used a beam projector (Optoma, PR320) with a size of 15 cm × 14 cm × 7 cm (X, Y, and Z axes, respectively) and two mirrors. Beam splitting [D3] used a customized beam splitter inside the surgical microscope system. The main functions of divisions [D2] and [D3] are to back-project the acquired PA B-mode image onto the surgical microscopic view plane through the ocular lens.

Results and discussion

A carbon fiber with a diameter of ∼ 6 μm was imaged in water at a depth of 1 mm using the NIR-VISPAM (Fig. 2a-c) to quantify spatial resolution. The cross-sectional PA B-mode and PA maximum amplitude projection (MAP) images of the carbon fiber are shown in Fig. 2a and b. Fig. 2c shows the axial and lateral PA profiles. The measured axial and lateral resolutions were 61 ± 1.4 μm and 36 ± 0.9 μm, respectively; these values are close to the theoretical resolutions of 57 and 32 μm, respectively.

Fig. 2

(a) Photoacoustic B-mode and (b) maximum amplitude projection images of a carbon fiber with a diameter of 6 μm, respectively. (c) Axial and lateral resolution profiles.

To show the feasibility of the NIR-VISPAM system, we guided a needle (27 gauge) and monitored the injection of carbon particles solution (carbon – glassy, spherical powder, Sigma-Aldrich) into chicken breast tissues containing a black polyvinyl chloride sheath target at a depth of 1.85 mm. The field of view (FOV) of the back-projected PA B-scan image was 10 mm × 13 mm (X × Z). We photoacoustically guided needle insertion and retraction toward the target (Video 1). At the same time, we successfully visualized the local injection of the carbon particles solution near the target. To increase the SNR, we averaged twice to display one PA B-mode image, so the image display rate was 23 Hz in this experiment. Simultaneous microscopic and PA B-mode images were screen-captured through the right ocular lens before, during, and after injection of aqueous solution of carbon particles using the needle near the implanted target in the chicken breast tissue as shown in Fig. 3a, c, and e. Fig. 3b, d, and f show the close-up images of the inset PA B-mode images in Fig. 3a, c, and e, respectively. As the video proved, the invisible NIR light did not disturb the operator's vision. The PA MAP images were acquired before (Fig. 3g) and after (Fig. 3h) injection of the carbon particles solution with a FOV of 10 mm × 10 mm along both X and Y axes. In Fig. 3h, the location of the target in the chicken breast tissue was deviated from the original location due to the needle intervention.

Fig. 3

Ex vivo real-time NIR-VISPAM by guiding needle insertion and displaying the injection process of carbon particle solution in a chicken breast tissue. (a), (c), and (e): Screen shots of overlaid PA B-mode and surgical microscopic images obtained through the right ocular lens before, during, and after injection of carbon particle solution using the needle into a target embedded in the chicken breast tissue, respectively (Video 1). (b), (d) and (f): Enlarged PA B-mode images acquired from (a), (c), and (e), respectively. The PA MAP images of the target in the chicken tissue (g) before and (h) after injection of carbon particles. PA, photoacoustic; MAP, maximum amplitude projection; and NUT, needle ultrasound transducer.

We conducted in vivo interventional experiments to investigate the feasibility of NIR-VISPAM in practical intervention. All animal experimental procedures satisfied the laboratory animal protocol approved by the institutional animal care and use committee of the Pohang University of Science and Technology. B16 melanoma cells (∼2 × 105) were injected subcutaneously into the left thigh of a Balb/c nude mouse weighing ∼20 g. Seven days after injection, the melanoma tumor had grown to a diameter of ∼ 4 mm. Before needle intervention, the mouse was anesthetized by hypodermic injection (20 μL) of a mixture of Zoletil and Rompun (3:1 ratio). The animal was placed on a customized animal stage which can be moved in the X, Y, and Z axes. The laser pulse energy on the mouse skin was approximately ∼ 51 mJ/cm2, which is much less than the ANSI safety limit (100 mJ/cm2) at λ = 1064 nm. We successfully guided the needle insertion and retraction toward the melanoma in vivo under the guidance of NIR-VISPAM (Video 2). Simultaneously, we photoacoustically monitored the local delivery of carbon particles within the melanoma. Microscopic and PA B-mode images were concurrently screen-pictured through the right ocular lens before, during, and after the carbon particles delivery using the needle within the melanoma as shown in Fig. 4a, c, and e. Fig. 4b, d, and f are the magnified PA B-mode images acquired from Fig. 4a, c, and e, respectively. Fig. 4g and h show PA MAP images were taken before and after delivery of the carbon particles solution. The boundary of the melanoma and distribution of carbon particles were clearly delineated. For in vivo experiments, three PA B-mode images were averaged to improve the SNR, so the image display rate was 15 Hz. Note that surrounding blood vessels were not clearly visible, possibly because the optical absorption coefficients of oxy- and deoxy-hemoglobins at λ = 1064 nm are only 1/20 to 1/40 as strong as at λ = 532 nm [23]. Additionally, the laser pulse energy used was only ∼50% of the safety limit. We believe that the surrounding blood vessels can be identified if the laser pulse energy is increased.

Fig. 4

In vivo real-time NIR-VISPAM by guiding needle insertion and displaying the injection process of carbon particle solution in a melanoma bearing mouse. (a), (c), and (e): Screen shots of overlaid PA B-mode and surgical microscopic images obtained through the right ocular lens before, during, and after injection of carbon particle solution using the needle into the melanoma in the mouse, respectively (Video 2). (b), (d) and (f): Magnified PA B-mode images acquired from (a), (c), and (e), respectively. The PA MAP images of the melanoma (g) before and (h) after injection of carbon particles. PA, photoacoustic; MAP, maximum amplitude projection; and NUT, needle ultrasound transducer.

Conclusions

We have developed NIR-VISPAM, which combines a conventional surgical microscope and a PAM system that uses a NIR laser (λ = 1064 nm). Compared to the previously-developed VISPAM [22], the current system has four advantages: (1) NIR PA excitation does not disturb the operator's vision. (2) The laser safety limit is five times higher in in the NIR region than in the visible region. (3) The specially-designed needle US transducer simplifies operation by eliminating the water tray. (4) The maximum image display rate is improved by a factor of more than 10 (i.e., 45 vs 2 Hz). The novel image display strategy based on augmented reality is another key feature for fast clinical translation. In this case, no computer display is necessary, and the convenience would be significantly enhanced. We successfully guided needle insertion and retraction in biological tissues and tumor bearing mice. We also monitored local delivery of carbon particles in both tissues and live animals. To extend our concept, we will focus on (1) adapting an aiming beam to visualize the correct scanning area; (2) developing a real-time image processing method based on a graphics processing unit [24]; and (3) integrating various optical imaging modalities including optical coherence tomography and fluorescence microscopy [25,26]. For quick clinical translation, pigmented melanomas can be accurately delineated and the removals of the melanomas can be simultaneously guided by the NIR-VISPAM system. Despite the low light absorption of hemoglobin at λ = 1064 nm, this wavelength is still suitable to visualize microvasculatures [27,28]. If we use a more powerful laser source, we believe that the microvasculatures can be visualized in the NIR-VISPAM images. For monitoring local drug delivery, there is no clinically approved contrast agents at this wavelength. Thus, the PA excitation wavelength should be switched to ∼ 800 nm, where clinically approved indocyanine green can be visualized. Therefore, we believe that our NIR-VISPAM system will become a crucial tool in neurosurgeries, ophthalmological surgeries, dermatological surgeries, and/or free autologous tissue transfers.

Conflict of interest

The authors declare that there are no conflicts of interest.