First in man in-situ augmented reality pedicle screw navigation.

BACKGROUND: Augmented reality (AR) is a rising technology gaining increasing utility in medicine. By superimposing the surgical site and the operator's visual field with computer-generated information, it has the potential to enhance the cognitive skills of surgeons. This is the report of the first in man case with "direct holographic navigation" as part of a randomized controlled trial. CASE DESCRIPTION: A pointing instrument was equipped with a sterile fiducial marker, which was used to obtain a digital representation of the intraoperative bony anatomy of the lumbar spine. Subsequently, a previously validated registration method was applied to superimpose the surgery plan with the intraoperative anatomy. The registration result is shown in situ as a 3D AR hologram of the preoperative 3D vertebra model with the planned screw trajectory and entry point for validation and approval by the surgeon. After achieving alignment with the surgery plan, a borehole is drilled and the pedicle screw placed. Postoperativ computer tomography was used to measure accuracy of this novel method for surgical navigation. OUTCOME: Correct screw positions entirely within bone were documented with a postoperative CT, with an accuracy similar to current standard of care methods for surgical navigation. The patient was mobilized uneventfully on the first postoperative day with little pain medication and dismissed on the fourth postoperative day.
CONCLUSION: This first in man report of direct AR navigation demonstrates feasibility in vivo. The continuation of this randomized controlled study will evaluate the value of this novel technology.

Background

Augmented reality (AR) is a rising technology gaining increasing application in medicine. By superimposing the surgical site and the operator's visual field with computer-generated information, it has the potential to enhance the cognitive skills of surgeons. One crucial task in spine surgery is pedicle screw placement, which bears the risk of neurovascular injury or insufficient screw hold in case of inaccurate screw placement. In order to improve safety and accuracy of screw placement, navigational tools such as optical navigation systems [1], patient-specific instrumentation [2], and even robotic-assisted pedicle screw placement [3] have been developed.

In the last years, substantial efforts have been made to introduce AR as a novel surgical navigation technology into spine surgery [4], [5], [6], [7], [8], [9], [10], [11], [12], [13], [14]. Although promising results have been achieved in feasibility studies, only a few methods demonstrated efficiency in patients [6,14]. The aim of our research was to develop a method capable of visualizing the planned screw trajectories by a computer-generated hologram directly on the real surgical situs, which would enable the surgeon to constantly reconcile the surgical task with the navigation information in an intuitive way.

By leveraging surface digitization and inside-out-tracking, we developed a radiation-free approach for the registration of the preoperative plan to the intraoperative anatomy with only an AR head mounted device (HoloLens 2, Microsoft, Redmond, USA) and a marker-equipped pointer [7]. In this manner, expensive navigation systems with external cameras may be replaced by an affordable surgeon-centered navigation approach, which does not suffer from line-of-sight issues.

After completing pre-clinical validation, the first-in-man randomized controlled trial for AR-based holographic surgical navigation of pedicle screw placement in spine surgery could be started. In the following case description, we report on the case of the first patient treated with “direct” holographic spinal navigation.

Case description

Approval by the local ethics committees (NCT04610411) and the national agency for therapeutic products (Swissmedic; EUDAMED reference number: 19-02-027424)) for using the technology as a medical device within a clinical trial was obtained.

A standard two-level lumbar fusion case was chosen for the first-in-man application on a 57-year-old patient with severe refractory lumbar back and left leg pain due to L5 nerve root radiculopathy. MR and CT images showed degenerative spondylolisthesis at L4/5 with facet joint effusions, consecutive spinal stenosis, and bilateral foraminal stenosis. Advanced degeneration was also detected at the level L5/S1 with almost completely collapsed disc height, intervertebral osteochondrosis (Modic Type 1), and facet joint osteoarthritis. Indication for fusion from L4 to S1 was given (Fig. 1). The patient gave informed consent to be treated with AR-based holographic surgical navigation.

Fig. 1

Preoperative images: (A) sagittal fat suppressed MRI (turbo inversion recovery magnitude (TIRM)) and (B) sagittal MRI (T2 sequence) demonstrating segment degeneration at L4/5 and L5/S1, (C) axial MRI (T2 sequence) at level L4/5 showing spinal stenosis, (D) lateral radiograph showing accentuated spondylolisthesis at L4/5 in standing position.

Surgical planning

Preoperative lumbar CT data with a slice thickness of 1 mm (SOMATOM Edge Plus, Siemens Healthcare GmbH, Erlangen, Germany) were acquired, from which a 3D triangular surface model of each vertebra was generated using commercial segmentation software (Mimics 19.0, Materialise NV, Leuven, Belgium). An in-house developed surgical planning software was used to plan pedicles screw insertion points and trajectories in 3D. The screws were visualized as cylindrical primitives, which were manually placed on the 3D vertebra models by a surgeon. The trajectories were planned along the anatomic pedicle axis, with the entry point at the intersection between the transverse process and superior articular facet (Fig. 2). The insertion points and trajectories were then parameterized as 3D locations and direction vectors, and used as navigation information.

Fig. 2

: Preoperative CT reconstructions with planned screw trajectories (yellow).

Surgical procedure

The surgical planning data was stored locally on the Hololens device, which was then prepared for surgery following a validated cleaning procedure. A trackable pointer and a clamp for fixation of a marker on a drill sleeve guide were additively manufactured using biocompatible polyamide PA2200 and sterilized in our institution using steam pressure (Fig. 3). The surgical procedure was performed under general anesthesia with the patient in the prone position. The dorsal structures of the spine, such as the spinous process, lamina, and transverse process, were exposed from the midline in a subperiosteal manner as usual.

Fig. 3

Navigation equipment: (A) 3D printed pointer with fiducial marker, (B) HoloLens 2, (C) drill sleeve guide with fiducial marker mounted on a 3D printed clamp.

Registration of the bony anatomy

After exposure, the pre-calibrated HoloLens device was placed on the surgeon's head. The surgeon controlled the navigation process with gestures and voice commands (Fig. 4). The pointing instrument was equipped with a sterile fiducial marker (Clear Guide Medical, Baltimore, MD, USA) and used to generate a digital representation of the intraoperative bony anatomy. To this end, the surgeon carefully followed the contours of the spinous process, lamina, and transverse process with the tracked pointer. Marker tracking was implemented using the Aruco library, which was adopted work with Hololens 2. After acquisition of the 3D point cloud of the bony surface, a previously validated and published [7] registration method was applied to superimpose the surgery plan with the intraoperative anatomy. The registration result was presented in-situ as a 3D hologram of the preoperative 3D vertebra model with the planned screw trajectory and entry point for validation and approval (Fig. 5). Registration was done separately for each vertebra.

Fig. 4

Surgeon with augmented reality head mounted device during navigation.

Fig. 5

3D hologram of the preoperative 3D vertebra model with the planned screw trajectories projected in situ after registration in order to be validated by the surgeon.

Navigation

The L4 and L5 screws were placed using an AR-based holographic surgical navigation without any fluoroscopic control. A conventional drill sleeve guide with a depth limit (∅ 3.2 mm No. 03.614.010, Synapse System, DePuy Synthes, J&J) was turned into an AR-trackable instrument by mounting it to a sterile fiducial marker using a sterile 3D-printed clamp (Fig. 3). The navigation was performed visually based on the drill sleeve's position and orientation, which was acquired in real-time with the HoloLens camera and Aruco marker detection [15]. The current Euclidean distance from the planned entry point and the angular deviation from the planned trajectory was in-situ visualized in millimeters and degrees, respectively.

Furthermore, the direction of trajectory deviation was visualized by three points forming a triangle: the first lying on the entry point, the second on the planned trajectory, and the third on the current trajectory (Fig. 6). After achieving alignment with the surgery plan, a borehole was drilled limited to 40 mm depth. The borehole was checked for pedicle wall perforation with a ball tip probe, before inserting blunt k-wires. Finally, cannulated 7 × 45 mm pedicle screws were inserted under K-wire guidance. S1 screws were inserted in a standard manner under anatomic orientation and lateral fluoroscopic control at the end of the screw insertion procedure to limit the experiment to only four screws, as this was a first-in-man procedure. The final screw position was checked by fluoroscopy, showing a satisfying result. Further steps, like decompressive laminotomy and intervertebral cage insertion, were done in a usual manner.

Fig. 6

Surgeon's view during navigation showing current deviation of entry point (3 mm) and trajectory (2°) in real time (for safety reasons shown here only in a cadaver sample).

Outcome

Postoperatively, the patient showed a complete reduction of leg pain and no further signs of radiculopathy. Correct screw positions entirely within bone were documented with a postoperative CT (Fig. 7). The 3D evaluation of the surgical accuracy based on a comparison between preoperative planning and postoperative CT revealed a mean 3D-summed angular deviation of 7.3 ± 3.6° for the trajectories and 3.5 ± 1.9 mm for screw entry points. The patient was mobilized uneventfully on the first postoperative day with little pain medication, and dismissed on the fourth postoperative day.

Fig. 7

Postoperative images: (A) axial CT at L5 and (B) axial CT at L4 showing adequate position of navigated screws without pedicle perforation, (C) antero-posterior and (D) lateral radiographs showing final spinal fusion construct.

Discussion

This is the report of the first-in-man application of a new fluoroscopy-free direct holographic surgical navigation technique with in-situ trajectory guidance. This is an essential step for the implementation of AR as the next-generation surgical navigation in surgery. Although surgical accuracy and user-friendliness have to be investigated with more cases, this case report proves the feasibility of direct holographic surgical navigation in an in-vivo setting.

State-of-the-art navigation technologies in spinal surgery have superior accuracy than the free hand technique [2,[16], [17], [18], [19], [20]]. However, the main limitations of such navigation systems are high set-up and maintenance costs, even if these systems might be cost neutral in the long-term in high volume centers [21,22]. This assumption is supported by a survey by Härtl et al., who revealed that surgeons cited high costs as one of the main reasons for not to use navigation systems [23]. Newer robotic assisted navigation systems are associated with even greater costs [24,25]. From technical viewpoint, a considerable limitation of “traditional” optical navigation systems is the dependence to an external camera system, which make it more difficult to have a clear view to the fiducial markers on the anatomy and surgical tools. According to a recent study, line-of-sight problem occurs multiple times in nearly every navigated neurosurgical procedure [26].

Another known limitation is the attention shift, which occurs when the surgeon is obligated to fix the gaze on a remote screen during navigation [27,28]. The approach presented here overcomes such limitations by combining a small, portable, and affordable device with computer vision software (Fig. 3).

However, the here reported novel method of navigation introduces new limitations: First, the operator needs previous training in order to be able to use the system reliably. In our experience, user-dependency seems to be higher at this stage compared to current standard navigations systems. Second, accuracy is dependent on the quality of the registration process and absence of patient motion. Accuracy is high in cadaveric experimental setting with up to 97.5% (unpublished data), comparable to current computer based navigation techniques (96% [16]) or even robotic assisted navigation (95-98% [17,29,30]) and certainly surpassing the conventional free-hand technique (43% to 86% [16,31,32]).

Compared to other navigation techniques, the here presented method seems advantageous, as the surgeon remains the last instance of quality control: He should be able to notice if the projected hologram is not aligned with the anatomy. Eventually, the currently running RCT will provide quantification of accuracy. Third, another potential limitation for broad clinical usage is the potential inconvenience associated with wearing a head mounted device. Further studies evaluating experience and surgeon's acceptance using this navigation are in progress.

So far, we found only a few clinical studies evaluating similar navigation technologies in humans [6,14]. Elmi-Terander and his group uses a sophisticated AR technology based on a video system with four-cameras, permitting fusion of 3D CT information with live video images of the surgical field [6,[33], [34], [35]]. Charles et al. investigated the same system and confirmed applicability in minimal invasive procedures [36]. However, the system of Elmi-Terander et al. is burdened with some degree of attention shift, since the surgeon is still obligated to fix his gaze on a remote screen for navigation. Molina et al. uses a Food and Drug Administration (FDA) approved AR navigation system with a head mounted device which projects navigation information directly into the operator's retina using a transparent near-eyedisplay [14]. In this way, the surgeon sees a 3D segmentation of the spine, overlaying the anatomy, and all navigation information displayed aside. Their approach is promising in reducing attention shift, but their registration method requires to acquire an intraoperative CT [14]. General application of such systems is limited due to the necessity of additional costly equipment. Therefore, we aim to provide an intraoperative image-free method of registration of anatomy. However, our approach without an anchored marker is yet prone to failure in case of position changes of the patient. We believe however that such an error can be noticed by the operator as an obvious offset of the hologram overlay on anatomy.

Conclusion

This case report presents the first in man application of a portable, fluoroscopy free AR based in situ navigation system. While this innovation overcomes some important disadvantages of the current navigation system, it introduces new challenges that need a careful incremental improvement process.

Funding Disclosure Statement: No financial funding sources were acquired for this case report. This work is part of «SURGENT», a flagship project of University Medicine Zurich/Hochschulmedizin Zürich.

Declarations of Competing Interests

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.