Stereoscopic liver surface reconstruction.

The paper presents a practical approach to measuring liver motion, both respiratory and laparoscopic, with a tool guided in the operating room. The presented method is based on standard operating room equipment, i.e. rigid laparoscopic cameras and a single incision laparoscopic surgery trocar. The triangulation algorithm is used and stereo correspondence points are marked manually by two independent experts. To calibrate the cameras two perpendicular chessboards, a pinhole camera model and a Tsai algorithm are used. The data set consists of twelve real liver surgery video sequences: ten open surgery and two laparoscopic, gathered from different patients. The setup equipment and methodology are presented. The proposed evaluation method based on both calibration points of the chessboard reconstruction and measurements made by the Polaris Vicra tracking system are used as a reference system. In the analysis stage we focused on two specific goals, measuring respiration and laparoscopic tool guided liver motions. We have presented separate examples for left and right liver lobes. It is possible to reconstruct liver motion using the SILS trocar. Our approach was made without additional position or movement sensors. Diffusion of cameras and laser for distance measurement seems to be less practical for in vivo laparoscopic data, but we do not exclude exploring such sensors in further research.

Introduction

Nowadays one of the most important directions in radiology and surgery is computer-aided diagnosis and therapy [1]. As regards computer-aided surgery, the research is focused on image-guided procedures. In order to construct the navigation system, information on movability is required, especially as regards the soft tissue in the abdomen.

Our objective was to develop a non-invasive methodology of liver surface points detection and tracking. There are many approaches relating to computer-assisted surgery, e.g. laparoscopic teleoperated robot-assisted surgery, da Vinci, image-guided laparoscopic surgery, and laparoscopic surgery computer simulators [2, 3]. Our approach was to use standard equipment (laparoscopic cameras, typical surgical tools), and not interfere with medical procedures. We also planned to use our approach for both laparoscopic and single incision laparoscopic surgery [4, 5].

While developing the methodology we took into account existing approaches. Many works have been done in beating heart surgery. Bogatyrenko et al. proposed a stochastic physical model of the beating heart surface [6]. Acquisition was done with a trinocular camera system [7]. Cortadellas et al. proposed the concept and overview of an augmented reality system that assists laparoscopic surgical procedures [8]. Mountney et al. proposed stereo cameras, a complex anatomical structure movement based on a modified Bouguet algorithm and used phantom tissue to model 3D geometry [9]. There also exists heart motion estimation and prediction using a Fourier series model, a vector autoregressive model and the extended Kalman filter. Groeger et al. proposed a laparoscopic tool tracking colour base segmentation and solved the problem of deformation and illumination occlusion of objects [10]. Chen et al. presented a feasibility system for guiding the robotic removal of the prostate that will provide augmented reality surgical guidance [11]. A stereoscopic fibroscope imaging system for minimally invasive surgery (MIS) is presented in [12]. The major goal was to design flexible instruments that can follow curved anatomical pathways by using the Simultaneous Localisation and Mapping (SLAM) algorithm.

An overview of the literature shows that the data about liver motion for a specific patient are not known so our future development work is regarded by us as a milestone. In the results of our work we want to use in real time a non-rigid registration algorithm of liver motion. This method focuses on simple algorithms which can be used in deterministic short time and are applicable to an online processing approach with a constant constrained frame rate in consultation with the surgeon.

Aim

The paper presents a practical approach to measurement of liver motion, both respiratory and laparoscopic, with a tool guided in the operating room.

Material and methods

Our approach can be divided into four steps: equipment setup, camera calibration, selecting the 3D reconstruction algorithm, selecting the triangulation points.

Equipment setup

The first and most important principle of our approach was to develop a non-invasive methodology (Figure 1). Laparoscopic cameras are nowadays standard equipment in the operating room, so we have focused on computer vision algorithms and we need at least two cameras to obtain a pair of stereoscopic images. However, there are some monocular reconstruction algorithms but there are many limitations which are difficult to take into account during in vivo laparoscopic reconstruction, e.g. requirement of a preoperative 3D model and ambiguity of depth map reconstruction. In order to apply a computer vision paradigm we need to mount the cameras together rigidly, so we prepared a special connecting device as seen in Figure 2. To set the system up we use two complete laparoscopic vision systems: Olympus CV-180 video processor HDTV, CLV-180 Light Source and an “Endo-Eye” laparoscopic camera with a Matrox Orion HD frame grabber. One important part of the setup is to synchronize the video stream. Generally trigger input is not included in the laparoscopic video acquisition system. This was the reason we needed to develop a software camera synchronization solution. This is based on an application with two background threads, and to balance them to record equal frames on both cameras. We use a global counter in the main thread to mark the frames. For laparoscopic surgery we use an Olympus LESS trocar.

Figure 1

Setup diagram

Figure 2

Stereo camera setup

Camera calibration

In our research we tested two calibration methods, the Tsai algorithm [13] and the Bouguet [14] method, both with satisfactory results. The Bouguet algorithm is based on a simplified pinhole camera model and low computation requirements. The Tsai method is more complex and can be divided into two steps: calculating the pose (3D orientation, and x-axis and y-axis translation) in the first stage, and computing the focal length, distortion coefficients and the z-axis translation in the second stage. We use the calibration grid and OPENCV Library (Hough Transform) to get sub-pixel coordinates of corresponding points and calibrate the camera with the Roger Y. Tsai algorithm [15]. For calibration we used two perpendicular calibration grids (chessboards). We gained the best results when the number of calibration points was greater than 30.

Selecting the 3D reconstruction algorithm

There are many approaches to the 3D reconstruction problem, but there are many limitations to its use in the laparoscopic reconstruction. Difficult cases with shape from shading (SfS) need multiple light sources [16, 17]. Because of liver surface reflection we use only one light source and the SfS is not practical for our approach. The texture of the liver surface is not distinct, so we also reject the shape from texture (SfT) [18] approach. After evaluation of different reconstruction approaches, we decided to use the classical triangulation technique. Despite the imperfections of the triangulation method, such as sensitivity to illumination and artefacts that cause pixel occlusions, we decided to adopt this method and prove its utility, and obtained high accuracy. Our approach is more challenging than one by attaching landmarks or colour markers to the liver surface. Triangulation τ is based on finding the position of a point in 3D with known camera matrices and projections of a point into both views (Figure 3). There are two types of estimation assessment: root-mean-square (RMS) residual error (distance of the measured and the estimated value) and RMS estimation error (distance of the estimated and the true value) [19]. We focused on open surgery, but our approach was flexible and could be used in laparoscopic surgery. Two rigid laparoscopic cameras were used.

Figure 3

The geometric interpretation of a triangulation of point X with known cameras C, C’

Selecting the triangulation points

As regards state of the art 3D reconstruction algorithms, the manual finding stereo correspondence seems not to be the perfect approach, but because of nonlinear shape changing in different segments of the liver, we decided to use it. Our main motivation was to verify the general knowledge of liver motion and collect specific knowledge to propose the proper shape deformation constraints to deploy the monocular shape from a motion approach (SfM) to the liver surface (based on standard laparoscopic equipment) [20]. Our preliminary approach was to use well-known solutions, i.e. RANSAC and Lucas-Kanade [21]. We tested Image and Video Processing – MATLAB and we were not satisfied with the RANSAC results. Secondly we evaluated the Lucas Kanade method in OPENCV for the optical flow of the liver. The reasons for the failure of both methods were lack of corners and edges, and a modest gradient on the limited visible part of the surface.

The fundamental problem of using the RANSAC algorithm and the other method (optical flow) for liver images is that the liver is not a rigid body. It is difficult to have a basis for the results obtained automatically for three reasons:

ruffles on the surfaces (some points of the surfaces disappeared in the frame sequence),

specific lightning caused by abdominal characteristics and the construction of the laparoscopic camera (after the pilot study we used only one light source to minimize the reflection of light effect),

liver deformation (the distance between specific corresponding points in the frame sequence is not constant).

Our data set consisted of 10 stereo video sequences for open liver surgery procedures. We also recorded 2 laparoscopic procedures using the SILS technique. After obtaining the videos from the cameras we tried to select corresponding frames using the global counter (Figure 4). During the video acquisition we used two balancing threads to grab frames from two cameras and mark frames with the value of the global counter. The steps of preparing data for the triangulation algorithm were:

Figure 4

Picture of the liver during surgery

select the corresponding frames based on the global counter values,

mark the characteristic points.

In order to assess the intra-operator variability in points of interest (POI), the manual clicking of the POI positions on selected stereo images was repeated twice by two operators. We tracked from five to ten POIs on each frame which were localized on specific liver segments. Basically we tried to use anatomical landmarks, but it was difficult, or even impossible. Therefore we decided to choose local extremes (Figure 5).

Figure 5

Zoomed pixel values – imshow and local extrema – red rectangles

In practice, we used imtool and imshow standard MATLAB image analysis tools (Figure 6). We also tried to use colour base segmentation. The motivation was that the colour of standard laparoscopic tools (silver or black) is different from the liver surface colour (hue of brown). Using this approach it was possible to find the contours of the laparoscopic tool, but impossible to find specific points. Because of our principle of using specific marked points, we discarded this approach.

Figure 6

Picture of the liver with the selected points of interest – imtool

The proposed evaluation method based on both calibration points of the chessboard reconstruction and measurements made by the Polaris Vicra tracking system are used as a reference system.

Results

We have presented separate examples for left and right liver lobes (Tables I-II). These results were obtained by performing the triangulation algorithm on manually found points in corresponding video frames. The first six left columns present the two different position of the same POI in a world coordinate system (measured in mm) in video sequence, the next two RMS residual error of reconstructed positions, and the last three the displacement vector. We obtained on average 3-4 mm RMS residual error with manually selected corresponding POIs. Due to vanishing pixels in the video sequence, RMS residual error of some POIs is up to 10 mm.

Table I

Sample data for right lobe laparoscopic tool guided motion

Initial position X i’			Final position X i’’			X i’ RMS error	X i’’ RMS error	Displacement ΔX i
X’	Y’	Z’	X’’	Y’’	Z’’			ΔX	ΔY	ΔZ
–371.12	134.23	–292.83	–362.02	115.8	–278.2	4.33	3.32	9.1	–18.43	14.64
–380.91	137.67	–300.91	–371.59	119.57	–282.64	4.33	3.41	9.32	–18.09	18.27
–376.38	139.8	–289.03	–370.87	116.39	–276.36	3.79	4.22	5.51	–23.41	12.67
–366.92	129.6	–287.85	–366.0	112.95	–275.24	4.08	2.82	0.92	–16.65	12.61
–365.85	128.02	–288.77	–369.23	112.35	–282.0	4.12	3.06	–3.38	–15.67	6.77
–360.16	129.03	–293.28	–361.99	112.16	–275.39	3.98	3.09	–1.83	–16.88	17.89

Table II

Sample data for right lobe respiratory motion

Initial position X i’			Final position X i’’			X i’ RMS error	X i’’ RMS error	Displacement ΔX i
X’	Y’	Z’	X’’	Y’’	Z’’			ΔX	ΔY	ΔZ
–379.29	119.29	–306.6	–375.94	121.42	–305.09	2.73	2.17	3.36	2.13	1.51
–354.81	124.02	–274.31	–350.31	126.84	–272.28	3.96	3.15	4.5	2.81	2.03
–472.88	106.5	–455.97	–475.86	105.01	–456.23	13.43	13.75	–2.99	–1.49	–0.27
–490.03	97.89	–460.29	–493.59	96.24	–460.85	12.67	12.78	–3.56	–1.65	–0.56
–490.25	100.01	–440.12	–492.79	98.54	–440.94	9.48	9.37	–2.54	–1.47	–0.82
–450.27	111.73	–414.62	–451.37	111.06	–415.04	9.8	9.71	–1.11	–0.67	–0.42
–450.19	111.06	–415.66	–451.36	110.35	–416.09	8.96	8.87	–1.16	–0.71	–0.43
–445.63	111.32	–414.03	–446.61	110.8	–414.27	7.83	7.84	–0.97	–0.52	–0.24

Discussion

Generally the results confirm the expectations of the scope of liver movement [22, 23]. Respiratory motion is one rank lower than the possible motion guided by the laparoscopic tool. It is possible to reconstruct liver motion using the SILS trocar but it is more complicated to set up the equipment using two single view cameras than with open surgery. Using one stereo camera (e.g. da Vinci Surgical System) will be a great simplification as regards camera calibration and mounting the camera in the SILS trocar, but this is not commonly accessible in operating rooms nowadays.

Finding the correspondence points manually requires deep image analysis, which allows observation of the wrinkles of the surface of the liver. Some points of the surface disappear in the frames sequence. It creates a challenge to use monocular SfM reconstruction even to find the correspondence points automatically.

The presented pilot study triangulation approach to liver surface tracking with manually selected correspondence based on two standard monocular laparoscopic cameras needs improved accuracy to allow medical application, e.g. tracking respiratory motion during radiotherapy. We are planning our future work in two directions. One is to find correspondence points automatically with sub-pixel accuracy. Therefore we will try to use a specific Olympus camera narrow band imaging which is the camera specific mode to enhance the vessels to aid finding the POIs. The second is to try to use only one camera for monocular surface reconstruction.

Our pilot study is a preliminary stage of development of a navigation system for treatment of unresectable liver tumour. The navigation system allows the surgeon to track the current position of surgical tools as regards the positions of anatomical structures during intervention; therefore tracking movement of anatomical structures is required. There are a few methodologies to find a solution. Bogatyrenko et al. [6] proposed a stochastic physical model for the similar problem of beating heart surface tracking. Mountney et al. developed stereo cameras, a complex anatomical structure movement based on a modified Bouguet algorithm and used phantom tissue to model 3D geometry [9]. Groeger et al. used a laparoscopic tool tracking colour base segmentation and solved the problem of deformation and illumination occlusion of objects [10]. All of them need special equipment to obtain positive results: stereo camera [9], colour laparoscopic tool [10] or complex mathematical models [6], that need to be adjusted to a specific patient. We have proposed a methodology based on standard equipment and does not require complex mathematical calculations. A disadvantage of our method is extracting correspondence POIs manually, but it could be overcome by using specific POI image descriptors or applying the monocular shape from motion algorithms.