Spine Surgery Assisted by Augmented Reality: Where Have We Been?

This present systematic review examines spine surgery literature supporting augmented reality (AR) technology and summarizes its current status in spinal surgery technology. Database search strategies were retrieved from PubMed, Web of Science, Cochrane Library, Embase, from the earliest records to April 1, 2021. Our review briefly examines the history of AR, and enumerates different device application workflows in a variety of spinal surgeries. We also sort out the pros and cons of current mainstream AR devices and the latest updates. A total of 45 articles are included in our review. The most prevalent surgical applications included are the augmented reality surgical navigation system and head-mounted display. The most popular application of AR is pedicle screw instrumentation in spine surgery, and the primary responsible surgical levels are thoracic and lumbar. AR guidance systems show high potential value in practical clinical applications for the spine. The overall number of cases in AR-related studies is still rare compared to traditional surgical-assisted techniques. These lack long-term clinical efficacy and robust surgical-related statistical data. Changing healthcare laws as well as the increasing prevalence of spinal surgery are generating critical data that determines the value of AR technology. © Copyright: Yonsei University College of Medicine 2022.

INTRODUCTION

In the 100 years since Roentgen discovered the X-ray in 1895, a “new kind of ray,” medical imaging technology has enabled clinicians to demystify the intact human body, which was previously impossible. The rapid development of imaging technology, which brought about an “industrial revolution” in medicine, was due to the desire of doctors to further improve the accuracy of surgery while reducing related complications.123 Over the decades, various radiographic techniques have appeared, including three-dimensional (3D) reconstruction from two-dimensional (2D) images.45 By improving surgeons’ ability to convert 2D to 3D images with new imaging technology installed with faster scan speeds and higher resolution image quality, radiation exposure on patients and staff can be reduced.6 Although technology has been developing rapidly over the past few years, we have never stopped innovating, and continue to do so today. Computer-aided surgery has become an integral part of modern surgical procedures, and a series of derivative technologies have emerged as a result of the combined efforts of computer scientists and clinical researchers, including augmented reality-based head-mounted displays (AR-HMD) such as the HoloLens glasses and the augmented reality surgical navigation (ARSN) systems.

AR, virtual reality (VR), and mixed reality nomenclature may initially seem strange and easily confused for one another, as they have similar technical aspects in virtual technology.7891011121314 VR usually consists of closed computer-generated digital information, and it completely replaces the real world to create an immersive visual experience. Conversely, AR provides an overlapping environment consisting of computer-generated images superimposed upon a real stereotaxic space. Therefore, it provides an awareness of real-world depth perception. As early as in 1968, Ivan Sutherland, who was dubbed “the innovator of computer graphics and AR,” developed the first HMD system (The Sword of Damocles), which converts a plane line into a 3D form. In 1997, Peuchot first reported the use of AR in spinal surgery. His group designed an approach to generating AR, based on the VR tools, for correcting scoliosis through 3D visualization that allows surgeons to observe vertebral body displacement through superimposed 3D transparent imaging.15 In 2016, Elmi-Terander reported the first spine cadaveric study of pedicle screw placement (PSP) using ARSN; and 2 years later, the same technique was adopted in the first prospective clinical study in a hybrid operating room.1617 In 2018, Microsoft introduced the U.S. Federal Drug Administration (FDA)-approved AR glasses called HoloLens, which adapts sensual and natural interface commands for preoperative surgical planning. Then, in 2019, Molina published a cadaveric proof-of-concept study of an augmented reality heads-up device (AR-HUD) called xVision, allowing surgeons to observe a virtual 3D model of a patient’s specific spine. Moreover, in 2021, the xVision Spine System developed by Augmedics was approved by the U.S. FDA (Fig. 1).1819

Fig. 1

Timeline illustrating the development of augmented reality historical pioneers and milestone events.

The development of spinal surgery AR techniques has recently introduced new questions and new potential avenues for research on whether this could be combined with other novel techniques, such as navigational or robotic systems, and whether these innovations could bring spinal surgery to the next generation.202122232425 Although the clinical application of AR guidance in spinal surgery is still in its infancy, some pioneers have already made tremendous inroads with impressive research findings.2627 Here, our team describes the currently available AR-assisted spinal surgery techniques and summarizes the workflows and outcomes.

METHODOLOGY

Search strategy and study selection

We conducted our systematic review using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We searched the PubMed, Web of Science, Cochrane Library, and Embase databases from their earliest records through April 1, 2021. We used the search terms “spine” OR “spinal” OR “vertebra” related to “augmented reality” OR “virtual reality” OR “mixed reality.”

Eligibility criteria and selection process

We conducted our literature search to identify original research articles published in peer-reviewed journals. Inclusion criteria were as follows: application of AR-related technologies in spinal surgery on phantom systems, cadavers, or clinical practice, reported in English. It is worth mentioning that, even though HoloLens glasses have been identified as mixed reality technology in the Microsoft official promotions, after reviewing the related article, we believe that its primary function in recent spinal surgeries presents in the form of AR. Therefore, we included the paper describing the application of HoloLens glasses in spinal surgery in our systematic review. Exclusion criteria were AR concept confusion, technology illustration without application results, and non-spine-related surgery.

Data collection process and data extraction

After excluding the duplicate records, two independent researchers (YT and MG) screened the titles and abstracts of the included articles. Then, these two same reviewers independently conducted full-text screening to identify articles based on the study inclusion and exclusion criteria. After a full-text screening review, the included papers were retained, and the data was extracted. Therefore, in this systematic review, we extracted data composed mainly of publication year, model type, application level, primary purpose, important conclusion, work systems, strengths and weaknesses, and new trends in AR.

RESULTS

A total of 1318 records were found after searching the database, and an additional six records were identified through other sources. After deleting duplicates, the articles were screened based on the titles and abstracts. Then, the articles considered to be eligible for this review were included. Next, we screened the full-text papers to exclude those with concept confusion with AR, non-spine-related surgeries, or only technical mechanisms without experimental results. Finally, 45 AR-related articles were included (Fig. 2). Table 1 illustrates the experimental model, application level, primary purpose, important conclusions, and working system in the included AR studies.

Fig. 2

Flowchart illustrating the selection of articles included in systematic review. AR, augmented reality.

Table 1

Studies on AR Navigation in Spine Surgery

Authors, yr	Model	Segments	Surgery	Purpose	Important conclusion	Working system
Luciano, et al., 20117	Simulator	Thoracic	Pedicle screw instrumentation	Haptic technology workstation	Part-task simulator demonstrates high potential in preliminary evidence as a training tool for thoracic PSP.	ImmersiveTouch software Virtual 3D volume of a human spine
Weiss, et al., 20118	Phantom	Lumbar	Spinal injection	Image overlay system	Image overlay facilitated accurate needle insertion and can broaden the scope of interventional MRI.	Semitransparent mirror and screen Standardized reproducible electromagnetic tracking system (Aurora, Northern Digital)
Abe, et al., 20139	Phantom Clinical	Thoracic Lumbar	Vertebroplasty	Virtual protractor AR system	VIPAR was successfully used to assist in needle insertion, and there was no pedicle breach or leakage of polymethylmethacrylate in clinical trials.	Head-mounted display (HMD) (Moverio, Epson) High-resolution web camera (C905 m, Logicool)
Fritz, et al., 201310	Cadaver	Lumbar Sacrum	Osseous biopsy	Image-overlay technology	94% of lesions were sufficient for pathological analysis and diagnosis.	2D, AR image overlay prototype system Clinical 1.5-T MRI (MAGNETOM Espree; Siemens Healthcare, Erlangen, Germany)
Fritz, et al., 201211	Phantom	Lumbar	Spinal injection	Image overlay system	All anatomic targets were successfully punctured. The average time for needle path planning and insertion was 55 s and 1 min 27 s, respectively.	2D, AR image overlay prototype system Clinical 1.5-T MRI system
Fritz, et al., 201212	Cadaver	Lumbar	Spinal injection	Image overlay system	The image overlay navigated system was technically accurate. Disc with an obliquity ≥27° may be inaccessible.	2D, AR image overlay prototype system Clinical 1.5-T MRI system
Elmi-Terander, et al., 201616	Cadaver	Thoracic	Pedicle screw instrumentation	Compare ARSN with freehand	ARSN was feasible and superior to the freehand technique. Except for vertebral body axial rotation, all morphometric dimensions were risk factors for more significant breaches when performed with freehand.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Kosterhon, et al., 201743	Clinical	Thoracic Lumbar	Osteotomy	AR-assisted navigation system	Without neurological deficits, the deformed vertebrae were successfully resected according to the preplanned resection planes. The spine was restored in near-physiological posture.	HUD (Pentero, Zeiss, Oberkochen, Germany) Optical navigation system (Kolibri2.0, Germany) 3D software (Amira, MérignacCedex, France)
Ma, et al., 201733	Phantom Animal	Lumbar	Pedicle screw instrumentation	AR-assisted navigation combined with the ultrasound	Experimental outcomes demonstrated that the proposed navigation system has acceptable targeting accuracy and radiation exposure.	IV overlay device. Optical tracker (Polaris, Northern Digital, Inc., Canada) Ultrasound device (DC-6 Expert II, Mindray, China)
Agten, et al., 201839	Phantom	Lumbar	Spinal injection	AR-assisted navigation system	97.5% of AR-guided needle placements were either perfect or acceptable without unsafe needle placements, and the time to final needle placement was substantially faster with AR guidance.	HMD: Microsoft HoloLens Segmentation tool (syngo. via, 3D Printing; Siemens Healthineers CT version 1.2.0, Erlangen, Germany).
Deib, et al., 201868	Phantom	Lumbar	Various	Optical see-through HMD	Percutaneous, vertebroplasty, kyphoplasty, and discectomy procedures were successfully performed by HMDs guidance, the key anatomic landmarks, and materials reliably visualized intraoperatively.	HMD: Microsoft HoloLens. Biplane angiography suite (Artis Zee, Siemens Healthcare GmbH, Forchheim, Germany).
Elmi-Terander, et al., 201853	Cadaver	Thoracic Lumbar	Pedicle screw instrumentation	AR-assisted navigation system	The overall accuracy of PSP was 89% (total: 18), the average navigation time was around 90 seconds, and the error angle was around 0.98°. There was no correlation between navigation time and accuracy.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Elmi-Terander, et al., 201917	Clinical	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	AR-assisted navigation system	The overall accuracy of PSP was 94.1% (total: 253). There were no severely misplaced screws and no occurrence of device-related adverse event.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Gibby, et al., 201969	Phantom	Lumbar	Pedicle screw instrumentation	AR-HMD navigation system	The HMD-AR technology projecting reconstructed 3D and 2D CT images can be accurately superimposed over the lumbar model and used to place pedicle screws.	HMD: Microsoft HoloLens OpenSight AR software (Novarad, American Fork, Utah, USA)
Urakov, et al., 201913	Cadaver	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Workflow caveats of Microsoft HoloLens	There were three major medical breaches and four major inferior breaches in the AR group; also, the author separately elaborated the caveats of workflow in AR-assisted PSP.	HMD: Microsoft HoloLens OpenSight AR software (Novarad, American Fork, Utah, USA)
Müller, et al., 202014	Cadaver	Lumbar	Pedicle screw instrumentation	AR-HMD combined with pose-tracking system	There was no significant difference in accuracy between AR-navigated and pose-tracking systems.	HMD: Microsoft HoloLens Pose-tracking system (fusionTrack500, Switzerland) CASPA (Balgrist CARD AG, Zurich, Switzerland)
Burström, et al., 201973	Cadaveric Animal	Thoracic Lumbar	Pedicle screw instrumentation	ARSN combined with the automatic instrument tracking system	97.4% screws were correctly placed without breaching the pedicle walls, and there was no difference between Jamshidi needle and high-speed drill in terms of accuracy or surgical time per pedicle.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Liebmann, et al., 201965	Phantom	Lumbar	Pedicle screw instrumentation	AR-HMD combined with surface digitization system	The specific navigation method achieved registration and tool tracking with real-time visualization without intraoperative imaging.	HMD: Microsoft HoloLens (Microsoft Corporation, Redmond, WA, USA)
Auloge, et al., 202020	Clinical	Thoracic Lumbar	Vertebroplasty	AR combined with an artificial intelligence system	There was no difference between the accuracy of the AR group in the skin entry point and the trocar tip and fluoroscopy group; however, the time for trocar deployment was significantly longer in the AR group.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Carl, et al., 201941	Clinical	Cervical Thoracic	Tumor resection	Microscope-based AR navigation system	Microscope-based AR provided close matching of visible tumor outline and AR visualization in all cases, the mean percentage of HUD-AR use was 51%, and the switch time of HUD was 2 to 17.	HUD of operating microscopes Pentero and Pentero900 (Zeiss, Oberkochen, Germany) Microscope element software (Brainlab)
Carl, et al., 201940	Clinical	Cervical Thoracic Lumbar	Tumor resection	Microscope-based AR navigation system	The application of intraoperative CT combined with AR ensured high navigational accuracy (mean error around 1 mm), and low-dose intraoperative CT protocols reduced the 70% effective radiation.	HUD of the operating microscopes Pentero/ Pentero900 (Zeiss, Oberkochen, Germany) Anatomical mapping software (Brainlab, Germany)
Carl, et al., 201963	Clinical	Cervical Thoracic Lumbar	Various	Microscope-based AR navigation system	Identification of bony and artificial landmarks allowed validating registration accuracy, AR facilitated visualization of the target structures reliably in the surgical field, along with their surgical orientation.	The HUD of the operating microscopes Pentero and Pentero 900 (Zeiss,Oberkochen, Germany) Microscope element application (Brainlab)
Molina, et al., 201918	Cadaver	Thoracic Lumbar	Pedicle screw instrumentation	Comparative accuracy of AR with the conventional method	The accuracy of the AR system was superior to manual computer-navigated PSP, and the user experience analysis yielded “excellent” usability classification.	AR-HMD display (xvision; Augmedics, Chicago, IL, USA)
Edström, et al., 202050	Clinical	Thoracic	Pedicle screw instrumentation	Compare freehand and ARSN system in deformity	The procedure time of ARSN was not prolonged with significantly higher PS density in the construct. Pedicle density is significantly higher in the upper instrumented vertebra in ARSN.	ARSN (Philips Healthcare, Best, the Netherlands). Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT.
Elmi-Terander, et al., 202074	Clinical	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Comparative accuracy of ARSN with freehand	ARSN system demonstrated a statistically higher accuracy of PSP compared to the freehand technique, primarily spinal deformity cases. The proportion of cortical breach was twice in the freehand group than in the ARSN group.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Dennler, et al., 202037	Phantom	Lumbar	Pedicle screw instrumentation	Comparative analysis of the beginners with experience by freehand or AR navigation	The AR headset improved the precision of drilling pilot holes for PSP by non-experienced surgeons and primary drill pedicle perforation by 7.5% in the freehand group and 2.5% in the AR group.	HMD: Microsoft HoloLens 3D triangular surface model (Siemens Healthineers, Erlangen, Germany)
Hu, et al., 202049	Clinical	Thoracic Lumbar	Vertebroplasty	AR-assisted navigation system	AR had less frequency of fluoroscopy and shorter operative time during entry point identification and local anesthesia. Also, it had a more significant proportion of “good” entry points.	Planar-based calibration system Industrial camera (XCDSX90CR, SONY, Japan) Projector (DLP W7000, BENQ, Taiwan)
Balicki, et al., 202052	Cadaver	Thoracic Lumbar	Pedicle screw instrumentation	Robotic guidance combined with ARSN system	A fully integrated robotic guidance system can improve workflow and provide all clinical acceptable pedicle screw guidance with less than 2 mm of targeting error.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Edström, et al., 20204	Clinical	Thoracic	Pedicle screw instrumentation	ARSN system in different spinal procedures	The ARSN can perform highly accurate surgery, decreasing the risk for complications while minimizing radiation exposure to the staff. The workflow for ARSN preparation only occupied 8% of the total surgical time.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Burström, et al., 202075	Cadaver	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Robot-guided system for semi-automated pedicle screw	The system provided a clinically acceptable level of PSP compared to ARSN without robotic assistance. Also, the technical accuracy was superior to their own previously reported ARSN data.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Nguyen, et al., 202021	Phantom Clinical	Cervical Thoracic Lumbar	Pedicle screw instrumentation	Machine vision image-guided system	The system’s magnification increased the possible angle sensitivity of pedicle screw angle placement, executing screw insertion trajectories with more acceptable precision and increased control.	Operation light head of the 7D MvIGS system (installed with IR tracker and the stereoscopic cameras) Two embedded cameras
Liu, et al., 202070	Phantom	Lumbar	Pedicle screw instrumentation	Comparative analysis of the AR–guided compared to fluoroscopy	AR-guided percutaneous lumbar PSP was acceptable and more efficient than radiograph-guided placement, and the automatic-alignment method was as accurate as of the manual method, but more efficient.	HMD: Microsoft HoloLens (Microsoft Corporation, Redmond, WA, USA)
Carl, et al., 202062	Clinical	Cervical Thoracic Lumbar	Various	Microscope-based AR navigation system	Automatic image registration by intraoperative CT combined with the non-linear registration of preoperative image data ensured a high visualization accuracy that had been successfully applied in all cases.	HUD of the operating microscopes Pentero or Pentero 900 (Zeiss, Oberkochen, Germany)
Edström, et al., 202076	Clinical	Thoracic Lumbar	Pedicle screw instrumentation	Evaluate the staff and the patient radiation exposure in ARSN system	The low-dose protocol used for the final 10 procedures yielded a 32% effective doses reduction per spinal level treated, and the study demonstrated significantly lower occupational doses compared to previous reports.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Burström, et al., 202051	Cadaver Clinical	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Frameless reference marker system for patient tracking	The mean technical accuracy of the frameless marker system was 1.65±1.24 mm, and there were no statistical differences in accuracy between pedicle devices spanning up to seven vertebral levels.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Gu, et al., 202067	Clinical	Thoracic Lumbar	Pedicle screw instrumentation	Comparative efficacy of AR with the conventional method	The AR group showed minor bleeding, shorter operation time, and better ODI and VAS scores with fewer postoperative complications.	HMD: Microsoft HoloLens (Microsoft Corporation, Redmond, WA, USA)
Xu, et al., 202036	Phantom	Thoracic Lumbar	Pedicle screw instrumentation	Spatial AR-based surgical navigation system for PSP	The accuracy of the pedicle screw insertion point on the skin was 0.441±0.214 mm, the average time of the AR navigation system was around 14.1 mins, and the system avoided the use of glasses.	Spine surgical treating planning system Camera-projector system
Gibby, et al., 202022	Phantom Clinical	Lumbar Sacrum	Spinal injection	Percutaneous image-guided spine procedures using AR	OpenSight AR provided a direct visualization with a high degree of anatomical accuracy. Also, it decreased the procedure time and reduced exposure to ionizing radiation for stuff.	HMD: Microsoft HoloLens OpenSight AR (Novarad, American Fork, UT, USA) NOVAPACS (Novarad, American Fork, UT, USA
Peh, et al., 202077	Cadaver	Thoracic Lumbar	Pedicle screw instrumentation	Comparative accuracy of ARSN with fluoroscopy	The overall accuracy of PSP with ARSN was 94% compared to 88% for fluoroscopy, and there were no unsafe screws in the scoliotic cases by the ARSN system without radiation exposure.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
Buch, et al., 202166	Clinical	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Optimized pipeline installment in intraoperative holographic models of patient landmarks	The intraoperative pipeline was successfully employed to generate patient-specific holographic models, and the registration accuracy dramatically improved with optimization of pipeline and technique.	HMD: Microsoft HoloLens Segmentation software (ITK-SNAP v.3.6) StealthStation Neuronavigation (Medtronic Sofamor Danek, Memphis, TN, USA)
Molina, et al., 202164	Cadaver	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Evaluate the clinical accuracy of AR-mediated spine surgery	The overall clinical accuracy was 99.1%, and 99.12% implants were noted to be Gertzbein-Robbins grade A or B. Precision analysis of the inserted pedicle screws yielded a mean screw tip linear deviation of 1.98 mm.	AR-HMD (xvision; Augmedics, Chicago, IL, USA). CT-integrated table (AIRO, Brainlab) Medical Image Interaction Toolkit (MITK; Germany)
Molina, et al., 202119	Clinical	Lumbar Sacrum	Pedicle screw instrumentation	Evaluate the clinical accuracy and technical precision of AR-mediated	All six screws were Gertzbein-Robbins grade A without perioperative complications. The clinical trial showed no difference compared to cadaveric data. None of the surgeons reported difficulty in navigating views.	AR-HMD (xvision; Augmedics, Chicago, IL, USA) Intraoperative CT scan (O-arm; Medtronic, Ireland) Medical Image Interaction Toolkit (MITK; Germany)
Burström, et al., 202123	Clinical	Thoracic Lumbar Sacrum	Pedicle screw instrumentation	Compare the intraoperative CBCT scans to postoperative CT scans	Intraoperative CBCT with the ARSN system is reliable for ruling out pedicle screw breaches and can be used for intraoperative breach detection and revision, making routine postoperative CT scans unnecessary.	ARSN (Philips Healthcare, Best, the Netherlands) Hybrid operating room: maquet table, motorized ceiling-mounted C-arm system, 3D Cone Beam CT
von Atzigen, et al., 202126	Phantom	Lumbar Sacrum	Rod bending	Marker-less surgical navigation to reconstruct 3D pedicle screw head positions	The machine learning-based proof-of-concept achieved better accuracy compared to the benchmark navigation approach requiring contact with the anatomy while requiring less time to acquire the screw head position.	HMD: Microsoft HoloLens Segmentation software (Mimics version 19.0, Materialise NV, Leuven, Belgium)
Molina, et al., 202171	Clinical	Thoracic Lumbar	Osteotomy	ARMSS	Osteotomy execution was successfully implemented to resect an en bloc-wide marginal of chordoma while avoiding a tumor capsule breach through a posterior-only approach using ARMSS.	AR-HMD (xvision; Augmedics, Chicago, IL, USA). BoneScalpel (Misonix) Integrated tracking camera

AR, augmented reality; ARSN, augmented reality surgical navigation; HMD, head-mounted displays; PSP, pedicle screw placement; VIPAR, virtual protractor with augmented reality; 3D, three-dimensional; 2D, two-dimensional; HUD, heads-up device; CBCT, cone-beam CT; ARMSS, AR-mediated spine surgery; ODI, Oswestry Disability Index; VAS, Visual Analogue Scale.

According to our search, AR-related articles have been published since 2011 (n=2); and related articles began to increase; and reaching a peak in 2020 (n=18). Starting in 2019, clinical application reports gradually increased, indicating that AR-related spinal surgery-assisted systems have acquired acceptable preclinical and clinical experimental results. This may be directly related to the recent rapid development of hardware and software in the AR field. The most prevalent AR-assisted surgical navigation system included in our systematic review was the ARSN system, which was installed in a hybrid operating room (n=14) and had an HMD (n=17). The Microsoft HoloLens accounted for a large percentage of HMD designs (n=12). Another type of AR-mediated surgery-assisted device was the HUD, which uses operating microscopes as its hardware tool. This technique has already been used in neurosurgery for intracranial surgery for a long time. Moreover, with the recent development of AR technology, spinal surgeons have attempted its use in different spinal surgeries such as injection, vertebroplasty, tumor resection, and rod bending. The most popular surgical application was pedicle screw instrumentation (n=28), and the main responsible surgical level were the lumbar (n=40) and thoracic (n=30) regions. The distinctive workflows utilizing the concept of AR are described below. The pros and cons of the mainstream AR-assisted spinal surgery systems are also discussed.

Pedicle screw placement

PSP requires an accurate skin location and operative trajectory, which differs in every patient with the angle and width of the pedicle.2829303132 Less experienced surgeons performing PSP usually use repeated intraoperative, anteroposterior, and lateral fluoroscopy for guidance, which increases radiation exposure and prolonging operation time, and also causes an increased rate of postoperative complications. For reducing these occurrences, Ma, et al.33 combined ultrasound-assisted CT imaging techniques with AR techniques, which utilized integrated photography and integrated videography to achieve an accurate 3D AR display of the spatial position, thus facilitating visual analysis of the images and improving the accuracy of the surgery.3435 This has overcome the deficiencies of spinal ultrasonic technology, such as partial coverage of the bone surface by the acoustic shadow. It also eliminated the registration errors caused by localized skin and soft tissue deformities. Xu, et al.36 reported an interesting experiment that ingeniously utilized the concept of AR and assembled relatively inexpensive and easily obtained devices (projector and camera) that allow for superimposed virtual navigational images visible to the naked eye. Dennler, et al.37 compared the outcomes of PSP performed by beginner and experienced surgeons with either freehand or HoloLens-assisted guidance. The results showed no significant inferences in the two expert groups. Nevertheless, the less experienced surgeons in the HoloLens group showed a significant decrease in screw perforation and improved precision of screw inclination angle compared to the manual group.

Elmi-Terander’s team first applied PSP with an ARSN system installed in a hybrid operating room, and successfully inserted 94 thoracic screws into a cadaver guided by the ARSN system. The percentage of spinal screws allowed in the ARSN group was much higher compared to conventional methods. Then, Elmi-Terander, et al.16 reported the world’s first prospective clinical trial using an ARSN system for PSP. Recently, they clinically evaluated the incidence of PSP cortical breach absences using ARSN, or a freehand method, and the freehand group showed twice as many incidences compared to the ARSN group.36 Burstrom described another report about ARSN systems, which showed a reduced number of hooks resulting in better long-term postoperative outcomes in deformity correction cases.4

Spinal injections and percutaneous biopsy

Degeneration and trauma to the facet joints are the decisive factors affecting the progression of arthritis and lower back pain. Previous studies have reported on the use of ultrasound for needle placement on facet joints. However, beginners may not obtain transparent images of bone and nearby structures through ultrasonography.3839 Fritz, et al.12 reported on an AR image overlay system that consisted of a semitransparent mirror that displayed the planned epidural injection trajectory. The patient’s body and virtual guidance images were infused and displayed on a semitransparent mirror, and the image did not move when the observer’s visual angle changed. However, the system may not work if the tilted-angle intervertebral disc was more than 27°. The same device was then used to perform a bone biopsy on four metastatic cadavers, and 93.8% of the specimens were sufficient for pathological evaluation.10

Tumor resection

Cerebrospinal fluid loss or spinal cord movement are hazardous as they may trigger brain displacement. Therefore, accurately locating the extent of the tumor and minimizing complications may present a challenge in spinal tumor resection. Carl, et al.40 used intraoperative navigation combined with microscopy-based AR-HUD systems to treat internal or external dural lesions. Virtual images of close real-time tumor boundaries and important nerve and blood vessel structures were observed with an optical microscope. The average error in this system was reported to be approximately 1 mm. Carl, et al.41 then evaluated the time ratio of AR by analyzing the video records obtained microscopically in 10 cases of intradural tumor surgery. The average time of AR-HUD during the operation was about half of the total display time using a surgical microscope.41 Meanwhile, the operator only switched the AR function on and off about five times. Therefore, this kind of AR-assisted surgery was shown to ensure complete resection of the lesion site safety, control soft tissue dissection precisely, and significantly reduce the rate of cerebrospinal fluid leakage intraoperatively.

Osteotomy

Osteotomy is frequently used for multiplane correction of spinal deformities. It is challenging to develop a preplanned osteotomy plane accurately to attain the expected correction angle.42 Kosterhon, et al.43 created a virtual surgical resection plane based on a congenital wedge-shaped half-pyramid, and exported the virtual image combined with navigation guidance to the surgical microscope. The virtual 3D contour display facilitated the visual control of the direction of the high-speed drill bit safely and avoided excessive resection of the posterior pedicles and laminas. Molina, et al.44 recently introduced another technique, which uses the newly released AR-HMD xvision Spine System for an en bloc chordoma spondylectomy, allowing the surgeon to visualize the contralateral surgeon’s operation progress with a tracked pointer.

Vertebroplasty

Vertebroplasty is a percutaneous treatment for osteoporotic fracture that can contribute to iatrogenic neurovascular injury or bone cement leakage, even under fluoroscopy guidance.45464748 Hu, et al.49 introduced a planar-based calibration system to superimpose the merged intraoperative 3D image created by the camera-projector system onto the actual anatomical structure to create an AR visualization. However, the image that the machine generates can easily block the surgical area, and the operator may also block the projected image. Soon after that, Abe, et al.9 reported on an AR-assisted virtual protractor system enabling holographic visualization of the planned osteotomy needle trajectory. This allows surgeons to observe a 3D virtual image from different angles by rotating the head. However, this system cannot be combined with intraoperative computed tomography navigation or be used in complex spinal cases.

DISCUSSION

For ARSN systems that rely on optical tracking without lead-based protective equipment, high-precision surgical trajectories can be created with the help of a built-in navigation system that can be used to augment the actual surgical field of view with 3D image overlays.1650 The camera in the ARSN system tracks from four different directions. A standard tracking procedure can be adopted as long as four cameras detect at least five skin markers. This can significantly prevent the loss of tracking from accidental movement or deep surgical interventions.16 Additionally, this kind of noninvasive adhesive skin marker will not be affected by breathing movements or the distance from the reference.51 However, due to the limited space between the detector and the patient, it is impossible to set an appropriate instrument spacing for obese patients.16515253 Moreover, increased surgical time is also a challenge. A good optimization of the operation workflow, such as scanning multiple segments at once, could significantly reduce the operation time.5455 In addition, for the AR-HUD system which is installed on the microscope, various virtual objects can be set up as translucent, contoured, or stereoscopic 3D patterns. These can be temporarily hidden to avoid excessive interference.565758 Also, the AR-HUD system does not require the operator to switch attention to the monitor, which can negatively impact hand-eye coordination.40596061 Currently, visual immersion HUD is insufficient, and 3D images must be more precisely merged with actual anatomy instead of simply projecting the image on top of the target area.6263

As for HMDs, such as the HoloLens, they can construct computer-generated 3D imagery holograms from a transparent visor projected in the real world. It seems that the most mature part of the AR technique has already been put to public use. However, a review of previous literature suggest that related clinical practices were not so optimistic.64 First, the series registration approach with the necessary bony landmarks is time-consuming. Second, the manual calibration of holograms and models is tedious due to inaccurate mobile hologram gestures or voice control.1065 Due to the aforementioned technique concerns, Buch, et al.66 introduced a generalizable pipeline for improving registration accuracy, and the results significantly reduced registration errors. Furthermore, the excessive hardware head weight caused operator discomfort after a long period of use.676869 In addition, shading of important anatomical structures by the translucent virtual image was also an unsatisfactory factor.4470 Moreover, we must consider the impact of radiation exposure with the techniques mentioned above.71 Fortunately, most of these technologies can rely on a single scan to perform an entire surgical procedure, or adopt a particular strategy to reduce radiation.5 For example, Carl, et al.41 found that the radiation exposure was reduced by about 70% using a low-dose protocol, and that the radiation exposure could be further reduced by restricting the scan range to the surgical area.

FUTURE PERSPECTIVES

AR-assisted spinal surgery comprises a newly emergent technology. As relevant literature becomes published and more attention is drawn, related hardware and software are also updated, providing an accurate registration process with low-latency connections. Furthermore, it should be possible to control the operative trajectory accurately, as well as detect and provide feedback on the distribution of abnormal blood vessels and nerves in advance. Meanwhile, this may standardize and optimize the workflow, formulating evaluation standards and quantifying the analysis of the results based on the actual situation. Today, artificial intelligence and robotic technique are proliferating and evolving. In a recent cadaveric experiment, a fusion technique in which the ARSN system cooperated with a highly flexible arm robot, was reported.72 Surgeons can use the instrument tracking feedback system to calibrate the robot without any human-controlled registration. This feedback system also reduces the risk of an inaccurate screw entry point location, and protects the robotic instrument from slipping off when the drill bit reaches the surface of the bone.

LIMITATIONS

Nowadays, various AR-enabled technologies are emerging without specific criteria for judging them. Meanwhile, the limited publications with few supporting randomized clinical trials and meta-analyses make it impossible to use statistics to explain the practical value of these techniques in spinal surgery. Nevertheless, we must recognize that it is essential not to rely solely on the grading or accuracy of clinical technology to finalize the value of emerging techniques. The value of AR will gradually expand with the long-term tracking of patients and the accumulation of surgical experience.

CONCLUSION

This systematic review presents an overview of AR technology from its earliest conception to the current evolution of spine surgery. It is feasible to incorporate this with other surgery-assisted technologies to enhance surgical radiological guidance and improve its efficacy in clinical treatment. The overall number of AR clinical applications is still limited. With the improvement of medicolegal processes and the advancement of surgical automation, it is undoubtedly true that the development of AR techniques will soon open a new era of spinal surgery.