Application of an RGBD augmented C-arm for minimally invasive scoliosis surgery assistance.

Minimally invasive surgeries (MISs) are gaining popularity as alternatives to conventional open surgeries. In thoracoscopic scoliosis MIS, fluoroscopy is used to guide pedicle screw placement and to visualise the effect of the intervention on the spine curvature. However, cosmetic external appearance is the most important concern for patients, while correction of the spine and achieving coronal and sagittal trunk balance are the top priorities for surgeons. The authors present the feasibility study of the first intra-operative assistive system for scoliosis surgery composed of a single RGBD camera affixed on a C-arm which allows visualising in real time the surgery effects on the patient trunk surface in the transverse plane. They perform three feasibility experiments from simulated data based on scoliotic patients to live acquisition from non-scoliotic mannequin and person, all showing that the proposed system accuracy is comparable with scoliotic surface reconstruction state of art.

Introduction

Minimally invasive surgeries (MISs) are gaining popularity as alternatives to conventional open surgeries due to a reduction of patient discomfort, pain and recovery time and overall treatment costs [1]. However, this type of surgery introduces a new set of challenges for surgeons: limited dexterity and tactile feedback, indirect visualisation through a monitor and an endoscopic camera, loss of depth perception and loss of the global perception of surgical site, which all result in a steep learning curve. One technique used for scoliosis MIS surgery is video-assisted thoracoscopic surgery (VATS) [2] that consists of introducing the tools and endoscope via the patient side, which avoids unaesthetic scar on the back, minimises trauma on the patient muscles and allows faster recovery. Fluoroscopic images are commonly acquired to guide pedicle screw placement and to visualise the effect of the intervention on the curvature of the spine. This makes the C-arm, the only available tool for scoliosis intra-operative imaging, and only shows the effect on the spine in the coronal and sagittal plane. However, cosmetic external appearance is typically the most important concern for patients, while correction of the spine and achieving coronal and sagittal trunk balance are the top priorities for surgeons [3]. Integrating the cosmetic appearance in the intra-operatively strategies of spine corrections chosen by the surgeon would maximise the satisfaction and outcome on both sides. For long now, the community has investigated patient trunk surface reconstruction in order to assess pre- and post-operatively the scoliosis deformation [4, 5] using visible structured light for computing the depth information and obtaining three-dimensional (3D) reconstruction. With the release in 2011 of low-cost infrared structured light RGBD cameras (RGB for colour and D for depth) such as Kinect 1, numerous works have moved toward RGBD cameras to perform trunk reconstruction pre- and post-operatively. Three techniques can be found in the literature: fusion of point cloud [6] from multiple cameras, Kinect fusion 3D surface reconstruction [7] and point cloud from single RGBD camera [8].

To quantify scoliosis deformities from trunk surface reconstruction, the community has developed metrics in the three anatomical planes: transverse, sagittal and coronal. Lately, Seoud et al. [9, 10] proposed a novel index to quantify trunk deformities. This new index allows a smooth and local description of the deformation at all trunk levels and not only at the deformity apex. The proposed index includes complementary measurements taken in the three planes: the back surface axial rotation (BSR), the trunk deviations in the coronal plane and in the sagittal plane. This index was already used to evaluate quantitatively the changes in trunk surface deformities after scoliosis spinal surgery to compare it with changes in spinal measurements [11]. The study showed that current surgical techniques perform well in realigning the trunk; however, the deformity correction in the transverse plane, measured by the BSR, proves to be more challenging. A real-time visualisation and quantification of the correction of the deformity in the transverse plane using the BSR metric during the surgery would enable the surgeon to adapt the strategy dynamically according to the patient response. To be used as assistive tool during surgery, the assessment of trunk surface deformation must bring minimal perturbation to the surgical workflow; this requires: (i) real-time acquisition, (ii) automated processing (minimum human intervention) and (iii) legible visualisation of the metrics. Most of trunk reconstruction works rely on spatial or temporal multiple camera acquisition. Using multi-camera acquisition during surgery is feasible, as this can be done in real time; however, it is cumbersome on a C-arm or prone to occlusion if mounted on the surgical room ceiling. Only one camera is a preferable setup, but with a real-time reconstruction, criteria not met by Reyes et al. [8].

Contributions: We present a feasibility study of the first intra-operative assistive system for scoliosis VATS surgery composed of a single RGBD camera affixed on a C-arm, which provides the real-time visualisation of the surgery effects on the patient surface in the transverse plane using the BSR index calculated from RGBD data. Via multiple experiments from simulated data to real-time acquisition on a person, we show that the accuracy of the proposed system is comparable with state of art of non-intra-operative scoliotic patient reconstruction in their ability to measure the small changes occurring on the BSR index.

Methodology

Setup

The system consists of a mobile C-arm (Siemens Siremobil Iso-C) on which one RGBD camera (Carmine Short Range Primesense) is affixed. During VATS surgery, surgeons can acquire anterior-posterior (AP) or lateral (LAT) images to assess, respectively, the coronal and sagittal Cobb angles. For this study, we suppose that the surgeon leaves the C-arm in vertical position (LAT view) when not used as it is less cumbersome. In this configuration, we place the camera at the middle of the C-arm C-curve. The setup is shown in Fig. 1. According to Newton and Perry [2], the surgeon is placed anterior to the patient that is lying on its side, with its back free of any draping making it visible by the camera placed posterior. Using the OpenNI library [https://github.com/occipital/OpenNI2.], the video image and depth image of the RGBD camera can be acquired at the frame rate of 30 FPS, the video image size is 1280 × 1024 pixels while the depth image is 640 × 480 pixels. The library also provides the mapping function Ω from the depth image to the video image.

Fig. 1

Setup with RGBD camera (red circle) placed at middle of the C-arm curve

Using the data acquired from the RGBD camera, the BSR metric is computed following the different steps explained in the pipeline in Fig. 2. We explain further those different steps.

Fig. 2

Pipeline of the assistive tool

Patient point cloud from RGBD data

From every incoming pair of video/depth images , we reconstruct the corresponding coloured point cloud in the depth camera coordinate system. For every pixel in the depth image of depth , we calculate the corresponding colour thanks to the mapping function Ω. The knowledge of intrinsic parameters K of the RGBD camera given by the manufacturer allows to obtain the 3D point of colour c corresponding to the 2D pixel (in homogeneous coordinates: ) using below equation: The focal length f and the principal point are parameters included in K and given by the RGBD camera manufacturer.

Patient-specific coordinate system

When measuring scoliosis deformation, in order to provide comparable intra- and inter- patient metrics, patient-specific coordinate system (PSCS) are commonly used [12]. A PSCS is generally defined by easily traceable anatomical landmarks. We define our PSCS by four anatomical landmarks , which are, respectively, the C7 vertebral prominence (VP), the posterior–superior iliac spine' midpoint (MPSIS) and the left and right SISs (LPSIS and RPSIS). The anatomical landmarks need to be identified once by the surgeon at the beginning of the surgery. When identified, trackable markers (in our case, augmented reality (AR) markers) are attached at the landmarks locations allowing them to be tracked all along the surgery automatically. Their location can be observed on the mannequin on Fig. 1 – right with label and Fig. 8 without label. Using the ArUco library [https://www.uco.es/investiga/grupos/ava/node/26.], we detect the markers centres in the colour image and then compute their coordinates in the depth image. Depth images are noisy and instead of relying on a single depth value, we use as depth value the average of the valid depth values around a local neighbourhood of the marker centre (on a disc of radius a quarter of the marker size). Thus, we finally compute the 3D points for the four anatomical landmarks using (1). Thanks to the four 3D anatomical landmarks positions, we compute the transformation from the PSCS to the depth camera coordinate system (D) using (2). In Fig. 1 – right, we show an example of PSCS, the X-axis (in red) is parallel to the line LPSIS to RPSIS, the Y-axis (green) parallel to the line MPSIS to VP and the Z-axis (blue) perpendicular to the back. At this point, we can transform the patient point cloud to the PSCS

Fig. 8

(Top) position taken as reference and (bottom) participant increasing posterior left-rib hump

Cross-section and scoliosis metric computation

Using the patient point cloud in the PSCS, we compute the BSR metric as defined by Seoud [11]. N cross-sections of thickness along the PSCS Y-axis are created with . To make our measurements comparable, we fixed the number of cross-sections to N = 100. Every point cloud point is assigned into its nth cross-section with if . For every cross-section, the 3D points are mapped orthogonal to the 2D-plane (X, Z), representing an outline of the back at this cross-section, as shown in Fig. 3 – top. We perform a cubic spline regression on the curve to smooth the outline. Then, the BSR curve, as shown in Fig. 3 – down, is extracted along the cross-sections by calculating the angle of the outline double tangent line (green line) to the X-axis.

Fig. 3

(Top) outline of the back curve at one cross-section and (bottom) BSR curve along the cross-section

Experiments and results

We perform several experiments to validate our approach, first using simulated depth images from real scoliotic patient models, then using real acquisition on non-scoliotic mannequin compared with 3D reconstruction, and finally we show qualitative results on a non-scoliotic person.

Evaluation on simulation based on real scoliotic patients

The first experiment aims at quantifying the error induced by the use of RGBD data for BSR calculation. We use complete (back and front) 3D models of scoliotic patients acquired by the INSPECK system before and after surgery, each model is already placed in its respective PSCS. To quantify the error induced by the RGBD data regardless of tracking error on the PSCS, we simulate depth images from the 3D models [our ground truth (GT)] and reconstruct the 3D point cloud corresponding to the simulated depth image, which is then a degraded partial view of the original 3D model. Both are placed in the same PSCS. The pipeline is visible at Fig. 4.

Fig. 4

Simulated point cloud from scoliotic patient data process (left – 3D model of scoliotic patient, middle – simulated depth and right – point cloud generated from simulated depth)

The simulated depth image is computed by constructing an octree on the full 3D model point cloud and by performing ray tracing from the simulated viewpoint. The first intersection with the octree of a ray originating from the simulated viewpoint for each simulated depth image pixel gives the depth at this pixel. For more realism, we add noise on the obtained depth value following the Gaussian model of the axial error on Kinect 1 type of camera given by Nguyen et al. [13]. For each six 3D patient models, we perform this operation at different poses whose rotation R is defined by Euler chained rotations around the X-axis and Y-axis, leading to with and a translation on the Z-axis from 70 cm to 1 m by step of 10 cm. In total, we have 196 poses per model. For every class of poses , we compute (3) the root-mean-square error (RMSE) on the sum of residuals on each pose p in the class, which is similar to calculate the RMSE error between the BSR measurements at all cross-sections j and poses p compared with GT The results for poses classified by angles are presented in Fig. 5, in this case, each class is composed of four poses varying by their distances to the patient.

Fig. 5

RMSE error for () angles at all distances, one colourbar by patient

Overall the graphs of the RMSE error are under , which is largely under the BSR typical error of measurement of reported by state of art work from Seoud et al. [9]. We observe that for , the RMS error increases for all patients, those poses are leaning toward the patient head and parts of the back surface are then occluded by the posterior rib hump typically prominent for scoliosis patients. Finally, the observation of the RMSE range for each patient shows that the influence of the angle is patient dependent, as the range is small for Patient 1 (), but ten times higher for Patient 3. Classifying the poses by distance to patient as shown in Fig. 6, we observe an increase with the distance, coherent with the axial depth noise property.

Fig. 6

RMSE error for all angles at the different distances for the pre-operative and post-operative cases of the three patients

This experiment shows that a single RGBD camera system has an acceptable sensitivity for BSR calculation, assuming no error on the PSCS tracking. The latter is taken into account in the next experiment.

Evaluation on real acquisition of non-scoliotic mannequin

This experiment is using live data from our framework. For evaluation, we use a static polystyrene mannequin with no scoliosis. We place four AR markers at the anatomical landmarks as described in Section 2.3 and place the mannequin at four different poses that are trackable and various in angulations. First, we test the reliability of our measurement by recording for each mannequin pose multiple BSR measurements (100 in our experiment). Using the intraclass correlation test, we compute the absolute agreement score between all the measurements for each pose. The reliability score for each pose is shown in Table 1. Then, we quantify the full error induced by our system including the PSCS tracking error by comparing with a GT in the form of a 3D reconstruction of the static mannequin back created from Kinect fusion algorithm (RecFusion [www.recfusion.net.]) requiring a 30 s acquisition and shown in Fig. 7 – right.

Table 1

Reliability score and RMSE error to 3D reconstruction for several poses

Pose	R_x(30)	0	R_y(30)	R_y(−10)
reliability score	0.885	0.953	0.881	0.971
RMSE error, deg	0.507	0.674	0.508	0.879

Fig. 7

(Left) point cloud of live acquisition and (right) non-real-time 3D reconstruction of non-scoliotic mannequin using Kinect fusion used for GT comparison

We manually segment the 3D landmarks on the 3D reconstruction and compute the PSCS for the 3D reconstructed mannequin. We then compare the BSR metric between the 3D mannequin and the BSR measurement computed with our system at the different poses during the reliability test by computing the RMSE error on the sum of residuals from all measurements and the 3D reconstruction BSR, using (3) with only one class per pose. The RMSE error for each pose is shown in Table 1.

The reliability score is above 0.85 for the four poses, which is considered as a good reliability [9]. This means that for one given pose, we will get the same measurement. The RMSE error is under 1° for the four poses, which is under the BSR typical error of measurement of reported previously. This experiment tested the full framework pipeline and shows that our system can be used for BSR calculation. Easy to setup, the use of AR marker has, however, limitations when the marker is inclined, it is not tracked anymore. The detection of the VP marker is in some poses complicated, e.g. for poses, where corresponding to the neck is further than the pelvis – the neck curve makes the marker not detectable. Robust to perspective change markers such as proposed by Birdal et al. [14] would help to overcome this issue. The sensitivity of our system to the marker detection error is also a subject that should be investigated.

Qualitative results with real acquisition of non-scoliotic person

Finally, we use our system on a non-scoliotic person deforming its back during the acquisition. With this experiment, we show that our system can track the progression of deformations in real time. The processing time per RGBD frame is <25 ms. We show in Fig. 8 two frames of the acquisition, the centre image is taken as a reference (green line on the BSR graph). On the right image, the person is increasing the posterior rib hump by bringing back the left shoulder, increasing the BSR angle at the spine top level in the negative values as it can be seen in the right image. We refer the reader to the supplementary video for the full acquisition sequence.

Discussion

Through our experiments, we have demonstrated that the proposed system shows promising results regarding accuracy, improving the state of art.

The next step is to validate our system by bringing it into the operating room, first as a non-interfering system to study the validity of the BSR metric within the clinical context as well as the hypothesis regarding the surgical context described in Section 2.1 on which our system is built. Existing works in the literature such as Navab et al. [15] have proven that camera integration on C-arm in the operating room (OR) is feasible. However, several challenges remain to be answered and investigated before the full integration in the scoliosis surgery workflow. The reference BSR is currently calculated from a posture taken at the beginning of surgery. A study concerning the reference positioning in surgery, either through its standardisation or its specific adaptation to patients (e.g. using deformable registration from pre-operative to intra-operative reconstruction) is necessary in the future in order to relate the intra-operative results to the pre-operative and post-operative results.

Finally, as the RGBD camera is affixed on the C-arm, the X-ray images can be fully integrated into the framework in combination with the 3D point cloud from the RGBD camera, first, for visualisation purpose such as Habert et al. [16], later by combining it with 3D spine reconstruction from X-ray image using the articulated model from Boisvert et al. [17], and finally exploring their mechanical interaction to suggest intra-operatively strategies of spine corrections based on trunk deformations.

Conclusion

To conclude, we have shown in this Letter the feasibility of the first intra-operative assistive system for scoliosis VATS surgery composed of a single RGBD camera affixed on a C-arm through multiple experiments from simulated data to real-time acquisition on a person. Our proposed system is able to measure the changes on the BSR index in the sensitivity range defined by state of art of scoliotic patient reconstruction.

Compliance with ethical standards

All procedures performed in studies involving human participants were in accordance with the ethical standards of the institutional and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards. Informed consent was obtained from all individual participants included in the study.

Supplementary Data

Click here for additional data file.