In vitro evaluation of stereoscopic liver surface reconstruction.

INTRODUCTION: Tracking abdominal motion of organs is an important factor in image-guided navigation systems. The paper presents the evaluation methodology of a practical approach to measure liver motion, both respiratory and laparoscopic, with a tool guided in the operating room. AIM: Evaluation of the methodology of a practical approach to measure liver motion, both respiratory and laparoscopic, with a tool guided in the operating room.
MATERIAL AND METHODS: The presented evaluation method is based on standard operating room equipment, i.e. laparoscopic cameras. We decided to use two rigid cameras to gain stereo in order to reconstruct characteristic points by triangulation. Our research aim was to survey the impact of three parameters on reconstruction accuracy: the number of calibration points, the imprecision of camera assembly, and the difference in resolution of images.
RESULTS: Three calibration chessboard configurations were tested. The reconstructed landmark positions and residual mean square errors were presented in three phantom poses: the reference position, translated position and rotated position.
CONCLUSIONS: The presented approach is a development of the previous work. Our research proved the importance of a rigid stereo camera system and the use of high definition image resolution for both stages, namely calibration and reconstruction.

Introduction

Tracking abdominal motion of organs is an important factor in image-guided navigation systems. The paper presents the evaluation methodology of a practical approach to measure liver motion, both respiratory and laparoscopic, with a tool guided in the operating room. The evaluation was based on the methodology described in the previous paper [1].

Aim

The proposed evaluation methodology can be divided into three stages: configuration of the stereo camera, camera calibration, solving of the correspondence problem. Our research aim was to survey the impact of the following three parameters on reconstruction accuracy: the number of calibration points, imprecision of monocular camera assembly, standard and high resolution images.

Material and methods

Reconstruction of point X in 3D space, called the world coordinate system, requires solving the correspondence problem from two monocular cameras [2]:

X = τ(P L, P p, x L, x P),

where τ denotes the triangulation algorithm, P L, P P are projection matrices of the left and right monocular camera, and x L, x P are coordinates of correspondence point in the left and right images.

The first step was to set up the equipment and conduct the calibration procedure. In our approach Tsai's algorithm was selected as a calibration method [3, 4]. We tested Tsai's method with different configurations of the calibration chessboards and different numbers of the calibration points. Both cameras should remain in the same relative position after the calibration procedure. The calibration procedure requires a few pose acquisitions of the calibration grid from both cameras, which is difficult during a laparoscopic procedure. It is impossible to use standard laparoscopic cameras to grab required poses directly in a patient position in the operating room. After taking out the cameras from trocars, cameras are reassembled in the same relative position to each other. Inaccuracy of this process influences projection matrices P, whose entities are decomposed as follows [2]:

P = K[R|t],

where: K – calibration matrix with internal camera parameters, R – rotation matrix in the world coordinate system, t – translation vector in the world coordinate system.

After calibration of a stereo camera and setting up the equipment (Figure 1) we focused on localization of correspondence points in the pair of monocular camera images.

Figure 1

Setup diagram

The accurate reconstruction from 2 monocular videos require two types of adequacy:

frame correspondence in time dimension,

point correspondence in the image from both cameras.

The best solution is obtained by hardware synchronization. We rejected this approach because the assumption is to use standard laparoscopic cameras which do not make it possible. The proposed solution is based on time stamps of a video frame sequence, using time stamps from the AVI file format and the maximum time-shift threshold approach. The evaluation was done on exactly the same equipment that is used in the operating room: medical video processor Olympus Exera II CV-180 with constant value frame per second parameter, which defines our maximum time-shift threshold. Pairs of frames whose time-shift is bigger than threshold are rejected.

In the presented approach [1] two well-known methods for solving the correspondence problem were taken into account:

RANdom SAmple Consensus (RANSAC) [3],

Lucas-Kanade [5].

RANSAC and Lucas-Kanade require landmarks to be obtained in the pair of corresponding images. For evaluation of possible triangulation accuracy, we used artificial markers attached to an abdominal liver phantom (Photo 1). The defined marker pattern, as a characteristic configuration of white and black boundaries of the image areas, was applied to locate markers’ positions. Selecting correspondence points in pairs in both images was performed manually.

Photo 1

Liver phantom with attached landmarks and its image positions

Results

The selected configuration of the calibration chessboard in our approach is presented below (Photo 2). Three calibration chessboard configurations were tested: both monocular cameras were calibrated on the left chessboard, both monocular cameras were calibrated on the right chessboard, the left camera was calibrated on the right chessboard and the right camera on the left one.

Photo 2

Calibration chessboards’ configuration with marked axes of world coordinate system

Generally we tested triangulation results on two data sets: left and right chessboard corners, liver phantom assigned landmarks.

The first data set allows an easy evaluation of results because we know the reference position of chessboards which define the world coordinate system (Photo 2). We use 30 mm chessboards with 81 corners. For the above-mentioned three calibration configurations the origin of the world coordinate system was fixed in the right corner of the left calibration chessboard (Photo 2). Moreover, we did the tested triangulation in SD/HD with 27, 45 or 81 points (which means: 3 upper rows of chessboard corners, 5 upper rows of chessboard corner rows and all chessboard corners respectively). The average residual mean square error (RMSE) in millimetres is presented below (Table I). There is anomaly in RMSE for 45 calibration points in HD resolution that is a little greater than for 27. However, this exception does not affect the tendency that with more calibration points we obtain higher accuracy (less RMSE).

Table I

Average residual mean square error of chessboard corner triangulation

	Camera configuration
Number of calibration points	Left chessboard				Right chessboard				Both chessboards
	Camera resolution				Camera resolution				Camera resolution
	Left		Right		Left		Right		Left		Right
	HD	SD	HD	SD	HD	SD	HD	SD	HD	SD	HD	SD
27									1.78	3.28		3.43
45									1.80	2.83		3.34
81	0.12			0.93	0.16			0.21	1.61	2.09	1.19	3.16

The reconstruction positions of liver phantom markers were tested in three positions in the high resolution mode: reference position, translated position, rotated position.

The reconstructed position and the average RMSE in millimetres are presented below (Tables II, III). The reconstructed liver phantom landmarks in three positions are presented (Photo 3).

Table II

Phantom landmarks’ reconstructed positions

Reference position				Translated position				Rotated position
X	Y	Z	RMS	X	Y	Z	RMS	X	Y	Z	RMS
–215.97	31.57	–143.12	3.70	–386.80	35.22	–164.98	6.08	–386.83	38.86	–164.06	2.95
–229.21	41.81	–107.13	3.51	–413.42	43.77	–138.69	6.14	–411.80	50.05	–135.08	2.11
–225.21	51.85	–144.67	3.86	–396.85	53.62	–172.18	6.03	–394.55	59.05	–169.41	3.04
–234.21	66.25	–130.41	3.78	–410.48	67.40	–161.56	5.95	–405.14	76.30	–155.43	2.82
–195.51	24.21	–199.28	3.49	–350.56	27.15	–213.25	6.05	–353.38	27.47	–214.23	4.55
–190.51	16.05	–214.37	3.39	–338.36	19.68	–223.04	6.31	–341.52	20.16	–223.99	4.63
–180.10	4.45	–227.70	3.78	–323.70	7.81	–231.55	5.65	–327.97	7.29	–232.78	4.74
–166.03	–6.80	–234.26	3.40	–309.22	–3.44	–234.71	5.62	–315.25	–4.49	–236.34	4.95
–215.11	49.48	–177.58	3.95	–376.12	52.86	–197.16	6.26	–374.32	56.72	–195.63	4.04
–208.76	46.16	–197.61	4.25	–363.25	48.77	–214.40	6.41	–360.82	52.42	–212.99	4.55
–199.26	40.25	–211.55	3.93	–349.77	42.49	–225.13	5.72	–349.07	43.83	–224.76	4.87
–178.64	24.52	–237.51	4.26	–321.72	27.04	–242.80	5.39	–323.79	26.39	–243.32	5.18
–222.44	66.60	–177.36	3.93	–381.76	69.51	–199.30	6.28	–378.14	74.86	–196.73	4.25
–212.14	66.13	–195.56	3.82	–368.51	68.19	–215.55	6.27	–365.49	72.32	–213.88	4.68
–206.10	61.79	–212.20	4.22	–356.32	64.34	–227.76	6.23	–354.00	67.30	–226.73	5.03
–194.04	54.02	–227.88	3.98	–339.67	56.68	–239.06	6.06	–338.45	58.30	–238.62	5.27
–184.74	45.09	–239.63	4.43	–327.30	47.25	–247.20	6.38	–327.71	48.06	–247.23	5.47
–222.32	85.05	–172.96	3.97	–385.12	86.50	–197.06	6.19	–380.12	93.51	–193.55	4.19
–207.98	81.98	–210.78	4.49	–359.53	83.68	–228.28	6.57	–355.47	88.11	–226.58	5.51
–198.54	75.41	–226.17	4.04	–345.17	77.61	–239.40	6.10	–341.77	80.77	–238.29	5.42

Table III

Average residual mean square error of phantom landmarks’ triangulation

Phantom position – average RMSE
Reference position	Translated position	Rotated position
3.88	6.07	4.21

Photo 3

The reconstructed liver phantom landmarks in 3 positions: reference position (white), translated position (red), rotated position (green)

The comparison of results (Table I) indicates the advantage of the calibration in High Definition Mode over the calibration in Standard Definition. The suggested number of chessboard calibration points for Tsai's method is more than 30 [2]. The performed tests highlight the decrease of RMSE for a higher number of calibration points, with one exception – discussed in the previous paragraph. More accurate results were obtained for calibration with one chessboard (left or right), where average RMSE is less that 0.5 mm, compared with calibrating cameras on different chessboards (in the worst case average RMSE is 1.61).

Applying two chessboards for calibration does not cause different results in RMSE for any of the chessboard corners.

Discussion

As regards the calibration chessboards setup by using one chessboard we assume that image noise is identically distributed in the observed coordinates, which generally is not true. In order to minimize triangulation RMSE in the whole reconstruction volume, it is advisable to select the set of calibration points from the whole volume. Since it is difficult to obtain, one calibration chessboard is used in practice [3]. There are some descriptions of research using a set of calibration grids instead of one grid [6]. We tested both approaches. The achieved results have less RMSE in the case of one chessboard. The impossibility to calibrate rigid configuration of two monocular laparoscopic cameras in the operating room (OR) patient position required reassembly of the camera setup, which caused RMSE increase in the case of two perpendicular chessboards. Taking OR circumstances into account, one calibration chessboard seems to be enough.

Tsai's algorithm is a well-known solution for a camera calibration problem. It requires at least 11 calibration points [2], but normally between 20 and 60 are used. We applied two perpendicular, 30 mm calibration chessboards with 81 corners and tested 21/45/81 calibration points in both high and standard definition modes. The high camera resolution mode has less RMSE than the standard one. This is especially important if one calibration grid is used to calibrate both cameras (differences in RMSE between the high and standard mode are significant). The experiments proved the important impact of image resolution on triangulation accuracy.

The triangulation approach for surface reconstruction requires solving the correspondence problem. Two well-known approaches, RANSAC [5] and Lucas-Kanade [7], require localization of landmarks in pairs of corresponding images. For possible evaluation of triangulation accuracy we use artificial markers attached to the abdominal liver phantom. Our approach, which is based on a defined marker pattern, as a characteristic configuration of white and black boundaries of the image area, could not find all instances of markers (Photo 1). The reason for the omitted markers was a large difference from the defined pattern. To reconstruct a more complex liver surface we plan to use a Time of Flight camera for open surgery [8].

The sensitivity of the triangulation approach is a relevant issue in investigation on 3D reconstruction [9]. Measurements of differences in the reconstruction position by rotation of one out of two cameras at a few degrees in the operating room may cause a significant difference in reconstruction position of the liver surface (Photo 3). This is a challenge in the laparoscopic case where generally it is impossible to calibrate cameras in the same position as used for triangulation (patient position). The best solution for this approach is to use one stereo laparoscopic camera. There is one available solution which is a stereo camera in laparoscopic teleoperated robot-assisted surgery, da Vinci, but the price and long learning curve for an operating team are crucial obstacles for common use. Nowadays there is a lack of cheaper and implemented medical solutions from competitive companies.