Handheld projective imaging device for near-infrared fluorescence imaging and intraoperative guidance of sentinel lymph node resection.

We propose a handheld projective imaging device for orthotopic projection of near-infrared fluorescence images onto target biological tissue at visible wavelengths without any additional visual aid. The device integrates a laser diode light source module, a camera module, a projector, an ultrasonic distance sensor, a Raspberry Pi single-board computer, and a battery module in a rugged handheld unit. It is calibrated at the detected working distance for seamless coregistration between fluorescence emission and projective imaging at the target tissue site. The proposed device is able to achieve a projection resolution higher than 314  μm and a planar projection bias less than 1 mm at a projection field of view of 58  ×  108  mm2 and a working distance of 27 cm. Technical feasibility for projective imaging is verified in an ex vivo model of chicken breast tissue using indocyanine green as a fluorescence agent. Clinical utility for image-guided surgery is demonstrated in a clinical trial where sentinel lymph nodes in breast cancer patients are identified and resected under the guidance of projective imaging. Our ex vivo and in vivo experiments imply the clinical utility of deploying the proposed device for image-guided surgical interventions in resource-limited settings.

Sentinel lymph node biopsy (SLNB) has become a widely accepted method of nodal staging for patients with clinically lymph-node-negative breast cancer., Compared with axillary lymphadenectomy, SLNB significantly decreases the rate of postoperative complications, reduces the cost of patient care, and improves patients’ quality of life after surgery. The National Comprehensive Cancer Network (NCCN) guidelines on breast cancer have recommended SLNB after preoperative systemic therapy for patients with surgical axillary stages I and II.

Near-infrared fluorescence (NIRF) imaging of indocyanine green (ICG) is an established technique for medical imaging and image-guided interventions. In the recent years, various NIRF imaging systems have been developed and clinically validated for sentinel lymph node (SLN) mapping in breast cancer patients. These systems rely on either screen display or in-situ projection of NIRF images. The screen display method requires an additional screen that challenges the already limited space in an operating room (OR). In addition, it distracts the surgeons by switching their fields of view between the screen and the surgical site. The in-situ projection method enables SLN mapping in a natural mode of visual perception. However, previously reported projective imagers are bulky and require manual calibration at the specific working distance for coregistration between fluorescence emission and projective imaging.

In this letter, we report a projective imaging device that integrates NIRF acquisition, visible image projection, and working distance detection in a handheld unit. The device uses a single-board computer for data acquisition and processing, a laser diode (LD) array as the excitation light source, and an ultrasonic sensor for working distance detection. Fluorescence emission from the surgical site is acquired by the device, calibrated based on the detected working distance, and projected back to the surgical site for surgical guidance. Technical feasibility of projective imaging is validated in an ex vivo tissue model. Clinical utility of image-guided SLN resection is demonstrated in breast cancer patients. To the best of our knowledge, this is the first report of SLN resection surgery guided by a handheld projective imaging device with automatic working distance detection and calibration.

Figures 1(a) and 1(b) show the engineering design and the working prototype of the proposed projective imaging device. The overall size of the device is , incorporating an excitation light source module, a detection module, a projection module, a single-board computer, and a 3D-printed cover. The light source module consists of five 200 mW LDs at a central wavelength of 760 nm. An ED1-C20-MD engineered diffuser (Thorlabs Inc., New Jersey) is placed at the output window of the light source, providing a uniform excitation illumination. The detection module consists of a 0.3-megapixel EO-0413M CMOS camera (Edmund Optics, Mainz, Germany) coupled with a 0814MM lens system (AZURE Photonics Co., Fujian, China) and two FF01-832/37-25 optical filters (Semrock Inc., New York). These filters are placed in front of and behind the lens, with their angles of incidence and cutoff wavelengths carefully designed to ensure appropriate fluorescence detection in the desired field of view (FOV). A URM37 V4.0 ultrasonic proximity sensor is used to detect the working distance between the device and the surgical site. The projection module is a miniprojector (Philips, Amsterdam, Netherlands) with its optical axis aligned in parallel with the camera. A Raspberry Pi 3b single-board computer (Raspberry Pi Foundation, United Kingdom) performs the parallel tasks of fluorescence image acquisition, working distance detection, system calibration, image processing, and visible image projection in multiple threads at an average latency of 593 ms. Based on our clinical observation, this latency does not induce significant motion blurring during an SLNB procedure. The latency can be shortened by using field-programmable gate array and by optimizing the imaging algorithms.

Fig. 1

A handheld projective device for intraoperative NIRF imaging: (a) schematic design of the projective imaging device, (b) photographic image of the working prototype, (c) illustration of the projection bias induced at different working distances, and (d) projective bias of the device before and after calibration.

The FOV of the projector varies from to within the range of working distance from 15 to 40 cm. Structural design of the handheld device ensures that the optical axes of the camera and the projector overlap in the horizontal direction. However, there is an angular difference between two optical axes in the vertical direction, leading to the vertical bias between the projected and the actual images of fluorescence emission. Therefore, calibration in the vertical axis is needed to ensure accurate mapping. The image processing algorithm also crops the nonoverlapped FOVs between the camera and the projector so that fluorescence emission can be projected to the surgical site with high fidelity.

Figure 1(c) shows the generation of the positional bias between the projected image and the actual field of surgery in the vertical direction. Let us denote as the width of the overlapped area between surgical FOV and projective imaging; as the number of the projected pixels within ; as the number of the camera pixels within ; and as the current working distance. and are pixel sizes for the camera and the projector, respectively, at the current working plane, where and are constants related to the camera and the projector. Thus, can be derived as

Since remains as a constant within the working distance range, the number of pixels for the width overlap between the camera and the projector is a constant. For an arbitrary working distance , the projection bias is linearly correlated with the working distance:

Let us denote as the number of pixels in width : where and are constants.

To calibrate the projective device, 10 sampling points are acquired at different working distances from 20 to 30 cm, and the images are adjusted to the desired sizes coincident with the actual scene. The values of the cropping parameter are acquired at 10 different working distances and fit into Eq. (3) in order to obtain the calibration factors and . To verify the calibration accuracy, 10 testing points in the working distance range from 20 to 30 cm are selected, and the calibration factors are applied based on their actual working distances. According to Fig. 1(d), the planar projection bias is controlled within 1 mm through the working distance ranges from 20 to 30 cm.

The achievable spatial resolutions for fluorescence imaging and projective imaging are characterized by a 1951 USAF resolution target, as shown in Figs. 2(a) and 2(b). Top images are the acquired and the projected images of the target. Limits of resolving power are determined by the largest group of the element pairs where the spacing pattern is no longer discernible. The 1951 USAF target test shows that the projective imaging device is able to achieve the spatial resolutions of 140 and for fluorescence imaging and projective imaging, respectively.

Fig. 2

System resolution and sensitivity test: resolutions of (a) fluorescence imaging and (b) projective imaging tested by a 1951 USAF resolution target, (c) fluorescence image of ICG samples at different concentrations, and (d) plot of versus .

To test the fluorescence imaging sensitivity of the proposed device, we dilute ICG in water at concentrations of 10, 7, 5, 3, 1, 0.75, 0.5, 0.25, 0.125, 0.0625, 0.05, 0.005, 0.001, 0.0005, and , respectively. Figure 2(c) shows the fluorescence images acquired from these samples at an excitation light power of and an exposure time of 330 ms. According to Fig. 2(d), our device is able to detect fluorescence emission with sensitivity at an ICG concentration of around 5 nM.

The technical feasibility of projective imaging to guide SLNB is studied in a chicken breast tissue model ex vivo, as shown in Fig. 3. In order to simulate the lymphatic vessel and the SLN, a PTEC tube () is inserted from the top of the chicken breast tissue until it reaches an inserting distance of 50 mm and an inserting depth of 13 mm. 0.1-mL ICG () in lipid emulsion is bolus injected in the tube to simulate the lymphatic flow. The ICG solution accumulates at the end of the tube and diffuses into the surrounding tissue area to simulate the lymph node. Fluorescence emission of ICG in the tube is acquired and projected back onto the ex vivo tissue. With intraoperative guidance of projective fluorescence imaging as shown in Fig. 3, an operator makes an incision on the tissue by a scalpel, and the surgical intervention continues until the simulated lymph node is localized and resected. After resection, the projective imaging device is used to confirm the complete removal of the fluorescence-labeled tissue target.

Fig. 3

Demonstration of the projection-guided SLN resection procedure in a chicken breast tissue model ex vivo. (a), (b) Tissue model observed by the projective navigation system before and after the bolus injection of ICG. (c) Fluorescence marked area found under the guidance of projective imaging. (d) Projective fluorescence imaging of the surgical site and the resected tissue sample. (e)–(g) Top and side views of the tissue sample show complete removal of fluorescence marked area.

The clinical utility of the projective imaging device is further demonstrated by a clinical trial at the First Affiliated Hospital of Anhui Medical University (Protocol No: 2017-08-05 AF/SC-08/02.0). Figure 4 shows a representative surgical case. Before the operation, 1% methylene blue (MB) solution and ICG solution are injected into four aliquots of breast tissue near the areola. MB is injected simultaneously with ICG to validate the accuracy of our device in SLN localization. Five to 10 min after injection, a clinical assistant holds the projective imaging device for surgical navigation, as shown in Fig. 4(a). First, the projection window of the device is set at the breast tissue area where ICG is injected. The ICG injection points and the lymphatic vessels are clearly visible by the projective imaging device, as shown in Fig. 4(b). Second, the projection window of the device moves along the identified lymphatic vessel until fluorescence emission disappears, at this position the surgeon makes the incision to expose deeper lymphatic vessel and follows the projective imaging guidance further to localize and resect the SLN, as shown in Fig. 4(c). The procedure is repeated until more SLNs are localized. Finally, the portable projective imaging device is used to confirm the fluorescence emission from the resected lymph node, as shown in Fig. 4(d). According to Figs. 4(c) and 4(d), the lymph node located by the projective imaging device is identical to that labeled by MB. The projective imaging device is portable and can be used by a surgeon for versatile imaging tasks without requiring the additional OR space.

Fig. 4

SLN resection surgery under the guidance of the portable projective navigation system: (a) surgery under the guidance of the portable projective navigation system, (b) lymph vessel observed by the projective navigation system after ICG injection, (c) SLN observed by the projective navigation system during resection, and (d) resected SLN.

In the current design of the projective imaging device, the ultrasonic sensor detects a single distance between the projector plane and the surgical plane, without considering the curvature of the actual breast tissue. The underlying rationale of this design is that the typical radius of a patient’s breast tissue is greater than 5 cm and the induced working distance difference on the surface of breast tissue will cause a positional bias of less than 2 mm in the projective image. For further improvement of the device in image-guided SLNB, a depth-resolved camera will be integrated for 3D topologic mapping of breast tissue and point-to-point calibration within the desired FOV. The total weight of the handheld device is 4.6 pounds, including the core components (2.4 pounds) and a 3D-printed cover (2.2 pounds). The overall weight of the device could be further reduced by design and manufacturing optimization.

Conclusion

In summary, we propose a handheld projective device for intraoperative NIRF imaging. In comparison with other NIRF imaging systems, the proposed device has advantages of portability, low-cost, high-flexibility, and simple operation. At a projection FOV from a working distance of 27 cm, the proposed system achieves a resolution of with a planar projection bias better than 1 mm. At a 330-ms exposure time, the projective navigation device is able to detect ICG concentrations around 5 nM. Technical feasibility of the device is verified in a chicken breast tissue model ex vivo. Clinical utility of the device is demonstrated by SLN resection surgery in breast cancer patients. Our ex vivo and in vivo experiment results imply the clinical potential of the proposed device for imaging-guided SLNB and for many other medical interventions where intraoperative guidance is necessary.