Needle tip force estimation by deep learning from raw spectral OCT data.

PURPOSE: Needle placement is a challenging problem for applications such as biopsy or brachytherapy. Tip force sensing can provide valuable feedback for needle navigation inside the tissue. For this purpose, fiber-optical sensors can be directly integrated into the needle tip. Optical coherence tomography (OCT) can be used to image tissue. Here, we study how to calibrate OCT to sense forces, e.g., during robotic needle placement.
METHODS: We investigate whether using raw spectral OCT data without a typical image reconstruction can improve a deep learning-based calibration between optical signal and forces. For this purpose, we consider three different needles with a new, more robust design which are calibrated using convolutional neural networks (CNNs). We compare training the CNNs with the raw OCT signal and the reconstructed depth profiles.
RESULTS: We find that using raw data as an input for the largest CNN model outperforms the use of reconstructed data with a mean absolute error of 5.81 mN compared to 8.04 mN.
CONCLUSIONS: We find that deep learning with raw spectral OCT data can improve learning for the task of force estimation. Our needle design and calibration approach constitute a very accurate fiber-optical sensor for measuring forces at the needle tip.

Introduction

Needle placement is a challenging problem for a variety of medical interventions, including brachytherapy or biopsy [12]. The force acting on the needle tip allows for inference about the currently penetrated tissue. This information can be used to navigate the needle and to prevent injuries of delicate structures [9]. In order to distinguish tissue based on tip forces, it may be required to measure those with an accuracy of approximately [8]. Tip forces cannot be measured with external sensors due to friction forces at the needle shaft [5]. Therefore, small-scale fiber-optical force estimation methods have been directly integrated into the needle tip. Several sensor concepts are based on Fabry–Pérot interferometry [1] and fiber Bragg gratings [6]. Here, we consider a setting where optical coherence tomography is available, e.g., to study tissue deformation [10] or to realize elastography [7]. While OCT has been proposed for tip forces estimation before [2, 3], these approaches rely on the reconstructed gray value data. However, using the reconstructed data has two limitations. First, the signal processing is based on a number of assumptions which may cause some loss of signal information. Second, it does not incorporate the phase part of the complex OCT signal that is particularly sensitive to small axial shifts. Therefore, we explore whether the tip force estimation accuracy can be improved by directly using the raw spectral OCT data. Thus, we perform a calibration between the optical signal and forces applied to the needle tip with convolutional neural networks. We validate our approach with three different needles using a new, improved needle design.

Methods

Needle design

We used an improved needle design for force estimation at the needle tip. A scheme and an image of the needle are shown in Fig. 1. A brass tip with a piston is put on a brass sleeve such that it is able to perform a sliding motion inside of it. An epoxy layer between tip and sleeve acts as a spring. An optical fiber is embedded into a ceramic ferrule. The ferrule is positioned relative to the tip piston such that the light beam travels a distance of approximately through air until hitting the surface of the piston. For protection, the ferrule is embedded into a polymer tube which is glued to the brass sleeve. When forces act on the needle tip, the epoxy layer is compressed and the piston moves closer to the exit point of the laser beam which can be detected in the OCT signal. The diameter of the needle is .

Fig. 1

The design of the needles used in this work. In the upper image, one of our needles is depicted below, in comparison with a standard G18 biopsy needle above. In the schema, the brass tip with a piston and the brass sleeve are depicted in yellow. The ferrule guiding the fiber is depicted in gray. The protection tube is depicted in orange

In [2], the needle tip was constructed as a cone and attached to the needle shaft with a deformable epoxy layer. Thus, radial forces on the needle tip could easily tilt it. The improved piston construction has the advantage that it guides the tip in axial needle direction and prevents tilt. This contributes to a more reproducible signal, an important aspect in the calibration of the needle, and to the overall durability.

Calibration data

We acquire calibration data for three custom build needles identical in construction. The data acquisition is performed similar to [3] where a needle is driven against a flat surface with a stepper motor. The force in the axial direction of the needle is measured with a force sensor and recorded together with the associated raw OCT data. Approximately 180 000 OCT-force pairs are collected for each needle for forces between and . We perform regular OCT data reconstruction, which includes the following steps: Throughout the reconstruction process, information can be lost due to the DC spectrum estimation strategy (2), the apodization (4) which eliminates high-frequency signal parts and the selection of the absolute signal value (6) as the final image. Figure 2 shows an excerpt of the collected data, both in raw (left) and reconstructed form (right).

Fig. 2

Example OCT image data over a period of time. Left, the raw OCT signal is shown. Right, the reconstructed A-Scans (depth profiles) are shown. One individual scan is depicted in one column. The scans were collected as part of a time series. Subsequent scans are adjacently depicted (M-Scan). The reconstructed scans allow human interpretation to some extent, e.g., the piston surface is visible. Patterns in the raw OCT data are much more subtle and not interpretable by the human eye

Dechirping the data by resampling and interpolating to new sampling points, based on manufacturer specifications

Estimation of the DC spectrum using an exponential moving average with a damping coefficient of

Subtraction of the DC spectrum

Apodization by using a Hann window for filtering the spectral data

Fourier transform for mapping frequency values to spatial values

Selection of the absolute signal value as the final reconstructed intensity image

The raw OCT signal has a size of where is the number of scans acquired over time. Reconstruction up to step (5) results in a complex signal which is also of size . Finally, by taking the absolute signal value, the A-Scan image sequence of size is obtained. This can be interpreted as a sequence of 1D depth images over time.

Deep learning architectures

We compare prediction performance for needle tip forces from both raw and reconstructed OCT data with different convolutional neural network (CNN) architectures. The considered architectures are variants of the ResNet [4], which is an extension of CNNs that enables better training through improved gradient flow. The ResNet models were originally built for 2D images with 2D convolutions. Here, the A-Scan data represent 1D images. Therefore, we adapt the architectures by replacing 2D convolutions by 1D convolutions. Furthermore, we replace the network’s original output layer for multi-class classification with a fully connected layer with one output for needle force regression. We consider several ResNet variants, resembling different network sizes, the regular ResNet34 and ResNet18 architecture as well as a smaller architecture with 2 residual blocks and 6 convolutional layers, which we name ResNet6. All deep learning models are implemented in PyTorch [11]. Learning is performed over 150 epochs with a batch size of and a learning rate of 0.005 using the Adam optimizer. As a loss function, we use the mean squared error which is defined aswhere y is the ground-truth force and is the predicted force value. We use of the data as a hold-out validation set. We performed five training runs with different random seeds and averaged the individual results.

Results

We report the mean absolute error (MAE) in mN between force predictions and force targets on the validation set. Additionally, we report inference times for the examined neural network architectures. All results are shown in Table 1. Figure 3 shows the relative differences in errors graphically. For most combinations of needle and model architecture, the error for learning on raw OCT data is lower compared with learning from reconstructed OCT scans. Particularly for the ResNet34 architecture, the calibration performance improves for all needles.

Table 1

Mean absolute error results in mN and inference times in ms

	Needle 1		Needle 2		Needle 3		Inf. times
	raw	recon	raw	recon	raw	recon
ResNet 6	8.54±0.14	7.22±0.14	23.10±0.40	17.15±0.33	9.02±0.08	11.29±0.09	1.11±0.00
ResNet 18	4.36±0.05	7.09±0.18	8.16±0.23	11.42±0.32	7.18±0.66	5.65±0.06	3.56±0.00
ResNet 34	4.40±0.06	6.61±0.19	6.95±0.24	11.14±0.32	6.08±0.06	6.37±0.05	6.43±0.00

Fig. 3

Bar plot showing the relative difference of the mean absolute error when learning from raw data

Discussion and conclusion

In this paper, we address the calibration problem of OCT-based needle tip force estimation using new and improved sensor concept with a piston and a guiding sleeve. In contrast to a previous OCT-based concept [2], the needle is not as sensitive to lateral forces by design while achieving similar calibration results with intensity data.

Furthermore, we study an approach for improving deep learning-based calibration performance even further. During OCT acquisition, a spectral signal is obtained which is typically reconstructed to an intensity depth image. We illustrate that learning is possible in an end-to-end fashion, i.e., the process of image reconstruction can be avoided. Moreover, the results improve in six out of nine setups, indicating that there may be information lost in the original processing that can be used when training the network on the raw signal. The proposed approach allows for precise estimation of forces at the needle tip, which is particularly interesting for force-based robotic needle placement. With typical robot control cycles of , a trade-off between accuracy and inference time must be balanced.

We find that learning forces from raw OCT data instead of reconstructed images work well, in particular, for larger deep learning models. For future work, our approach could be studied in more detail with additional deep learning methods and in different applications scenarios. Also, our approach could be extended to other applications where an imaging modality is used as a sensor signal, for example, in the context of motion tracking with OCT or ultrasound.