Navigation-Linked Heads-Up Display in Intracranial Surgery: Early Experience.

BACKGROUND: The use of intraoperative navigation during microscope cases can be limited when attention needs to be divided between the operative field and the navigation screens. Heads-up display (HUD), also referred to as augmented reality, permits visualization of navigation information during surgery workflow.
OBJECTIVE: To detail our initial experience with HUD.
METHODS: We retrospectively reviewed patients who underwent HUD-assisted surgery from April 2016 through April 2017. All lesions were assessed for accuracy and those from the latter half of the study were assessed for utility.
RESULTS: Seventy-nine patients with 84 pathologies were included. Pathologies included aneurysms (14), arteriovenous malformations (6), cavernous malformations (5), intracranial stenosis (3), meningiomas (27), metastasis (4), craniopharygniomas (4), gliomas (4), schwannomas (3), epidermoid/dermoids (3), pituitary adenomas (2) hemangioblastoma (2), choroid plexus papilloma (1), lymphoma (1), osteoblastoma (1), clival chordoma (1), cerebrospinal fluid leak (1), abscess (1), and a cerebellopontine angle Teflon granuloma (1). Fifty-nine lesions were deep and 25 were superficial. Structures identified included the lesion (81), vessels (48), and nerves/brain tissue (31). Accuracy was deemed excellent (71.4%), good (20.2%), or poor (8.3%). Deep lesions were less likely to have excellent accuracy (P = .029). HUD was used during bed/head positioning (50.0%), skin incision (17.3%), craniotomy (23.1%), dural opening (26.9%), corticectomy (13.5%), arachnoid opening (36.5%), and intracranial drilling (13.5%). HUD was deactivated at some point during the surgery in 59.6% of cases. There were no complications related to HUD use.
CONCLUSION: HUD can be safely used for a wide variety of vascular and oncologic intracranial pathologies and can be utilized during multiple stages of surgery.

augmented reality

arteriovenous malformation

cerebrospinal fluid

computed tomography angiography

extracranial–intracranial

heads-up display

intracranial–intracranial

magnetic resonance imaging

Intraoperative navigation and microscope integration are very useful, but can be limited when attention needs to be divided between the operative field and the navigation screens. A heads-up display (HUD), available in the aviation industry for many years, is a recent addition to the neurosurgery toolkit. HUD provides visualization of navigation information during the surgery workflow. There has been a limited use of HUD across the surgical field thus far. This study details our early use of HUD in skull base and vascular cases. To our knowledge, this is the largest series utilizing HUD for intracranial surgery.

METHODS

Institutional Review Board approval was obtained to perform this study. A waiver of consent was obtained to perform this retrospective review. This is a retrospective review of all patients who underwent intracranial surgery using HUD from April 2016 through April 2017. There were no other inclusion or exclusion criteria. Intraoperative navigation was performed with Brain Lab Curve™ Image Guided Surgery (Brainlab, Munich, Germany). The Zeiss Pentero 900 (Carl Zeiss Meditec Inc, Dublin, California) was used for the majority of cases, and the Leica OH6 (Leica Microsystems Inc, Buffalo Grove, Illinois) was used for only a small number of cases. Prior to surgery, preoperative imaging, usually contrast-enhanced MRI and CTA, were reviewed by the senior author (JB) and team. Using the Brainlab platform, the lesions of interest, surrounding vessels, and/or surrounding nerves/brain tissue were painted using the Brainlab Smartbrush function by a member of the surgical team and then approved by the senior author, JB. Patient registration and microscope integration were performed in the standard fashion. The operating room setup included the operating microscope, Brainlab Navigation, and Surgical Theater imaging (Surgical Theater, Mayfield, Ohio; Figure 1).

FIGURE 1.

HUD convention. Dashed lines represent the maximum dimension of the painted lesion. Solid lines represent the dimension of the lesion that is in focus at any given time. Therefore, dashed lines will not change in size, but solid lines will change depending on focal length. In addition, the object can be displayed as outline-only or filled in with different degrees of transparency.

The HUD could be overlaid at multiple time points during the surgery (ie, during exposure of skin, bone, dura, cortex, and/or lesion). Microscope integration was typically performed after dural opening at the time of first use, but could be done earlier in the operation if the surgeon wanted to use HUD for phases such as head/bed positioning, skin incision, craniotomy, or dural opening. By convention, HUD objects have either a solid or dashed outline (Figure 1). The dashed outline represents the greatest dimension of the object projected in the surgeon's point of view, irrespective of microscope focus depth. A solid outline represents the object dimension at the current focal depth. Objects could also be outlined or filled with different degrees of opacity.

Pathologies were grouped together as to their vascular, oncologic, or other origin. Lesions were labeled as superficial if they came to within 1 cm of the surface of the brain or calvarium and all others were labeled deep. Structures painted included the lesion itself as well as surrounding vessels, nerves, or brain/brain tissue. HUD accuracy was subjectively determined based on the visualized overlap of painted structures with the location of the actual structures. Accuracy was determined retrospectively by authors JB and JM and was graded as excellent (perfect overlay), good (minimal overlay displacement), or poor (significant overlay displacement). This assessment was an estimate, not a measurement. Accuracy was assessed when the painted objects first came into view. Other patient data were recorded by reviewing the medical record. The chi-squared test was used to compare accuracy at different depths with a significance level of 0.05.

HUD utility during phases other than lesion localization/resection was assessed for patients in the second half of the study by recording the other phases of surgery when HUD was used. Other phases of surgery included bed/head positioning, skin incision, craniotomy, dural opening, corticectomy, arachnoid incision, and intracranial drilling. Head positioning could only be performed prior to the start of surgery, but bed positioning could be performed either prior to or during surgery. In order to be counted for utility, the HUD had to be actively used during that phase of surgery (ie, not just be turned on). The decision to use HUD during a certain phase of surgery, however, was operator dependent. Additionally, it was recorded whether HUD was turned off during the case and the reason for deactivation.

RESULTS

Seventy-nine patients with 84 intracranial lesions were included in the study. One patient had 3 meningiomas, 1 patient had both an aneurysm and a meningioma, 1 patient underwent both an intracranial–intracranial bypass and an indirect bypass during a separate surgery, and 1 patient had both an aneurysm and an arteriovenous malformation (AVM). All other patients had a single lesion/procedure. Average patient age was 53.2 yr and 53.2% were female.

A wide variety of both cerebrovascular (28) and oncologic (53) lesions were evaluated (Table 1). Vascular lesions included aneurysms (14), AVMs (6), cavernous malformations (5), and intracranial stenosis (3). Oncologic lesions included meningioma (27), metastasis (4), craniopharygnioma (4), glioma (4), schwannoma (3), epidermoid/dermoid (3), pituitary adenoma (2), hemangioblastoma (2), choroid plexus papilloma (1), lymphoma (1), osteoblastoma (1), and clival chordoma (1). Other lesions included a frontotemporal abscess, an inferior frontal cerebrospinal fluid (CSF) leak, and a cerebellopontine angle Teflon granuloma years following a microvascular decompression. A transcranial surgical approach was used for 77 lesions, and a microscopic transphenoidal approach was used for 7 lesions (craniopharyngioma (4), tuberculum meningioma, pituitary metastasis, and clival chordoma).

TABLE 1.

Breakdown of Pathologies

	N	%
Cerebrovascular	28	33.3%
Aneurysms	14	16.7%
ICA	5	6.0%
ACA	3	3.6%
MCA	3	3.6%
PICA	2	2.4%
SCA	1	1.2%
AVMs	6	7.1%
CPA	2	2.4%
Cerebellar	1	1.2%
Parietoccipital	1	1.2%
Temporoparietal	1	1.2%
Lateral ventricular	1	1.2%
Cavernous malformations	5	6.0%
Temoral	3	3.6%
Cerebellar	1	1.2%
4th ventricular	1	1.2%
Intracranial stenosis	3	3.6%
Oncologic	53	63.1%
Meningioma	27	32.1%
Sphenorbital/clinoidal	10	11.9%
Convexity	9	10.7
Tuberculum	2	2.4%
Foramen magnum	2	2.4%
Tentorial	2	2.4%
CPA	1	1.2%
Torcular	1	1.2%
Metastasis	4	4.8%
Craniopharyngioma	4	4.8%
Glioma	4	4.8%
Schwannoma	3	3.6%
Epidermoid/dermoid	3	3.6%
Pituitary adenoma	2	2.4%
Hemangioblastoma	2	2.4%
Choroid plexus papilloma	1	1.2%
Lymphoma	1	1.2%
Osteoblastoma	1	1.2%
Clival chordoma	1	1.2%
Other	3	3.6%
Abscess	1	1.2%
CSF leak	1	1.2%
CPA Teflon granuloma	1	1.2%

There were 59 deep and 25 superficial lesions (Table 2). Structures identified with HUD included the lesion itself in 81 cases, surrounding vessels in 48 cases, and surrounding nerves or brain tissue in 31 cases. HUD accuracy was deemed to be excellent in 71.4% of cases, good in 20.2% of cases, and poor in 8.3% of cases. Deep lesions were less likely have excellent accuracy compared to superficial lesions (64.4% vs 88.0%, P = .029; Figure 2).

TABLE 2.

HUD Results

Location
Deep	59	70.2%
Superficial	25	29.8%
Accuracy
Excellent	60	71.4%
Good	17	20.2%
Poor	7	8.3%
Structures
Lesion	81	96.4%
Vessels	48	57.1%
Nerves	31	36.9%

FIGURE 2.

Accuracy analysis by depth. Deep lesions were less likely have excellent accuracy compared to superficial lesions (64.4% vs 88.0%, P = .029).

Utility was assessed for 52 (61.9%) lesions (Table 3). HUD was actively used during bed/head positioning (50.0%), skin incision (17.3%), craniotomy (23.1%), dural opening (26.9%), corticectomy (13.5%), arachnoid opening (36.5%), and intracranial drilling (13.5%). HUD was disabled in 59.6% of cases for a variety of reasons. Reasons to turn HUD off included that it was no longer useful (48.4%), distracting (38.7%), inaccurate (9.7%), or there was a technical difficulty (3.2%). We found associations between certain pathologies and measures of utility (Table 4). Examples of each measure of utility are demonstrated in Figures 3-12 and and . There were no complications related to HUD inaccuracy or over-reliance.

TABLE 3.

Measure of Utility

Utility
Lesions Evaluated	52	61.9%
Bed positioning	26	50.0%
Skin incision	9	17.3%
Craniotomy	12	23.1%
Dural opening	14	26.9%
Cortical incision	7	13.5%
Arachnoid dissection	19	36.5%
Intracranial drilling	7	13.5%
Turned off
Yes	31	59.6%
No	21	40.4%
Reason to turn off
No longer needed	15	48.4%
Distracting	12	38.7%
Inaccurate	3	9.7%
Technical difficulty	1	3.2%

HUD utility during phases other than lesion localization/resection was assessed for patients in the second half of the study by recording the other phases of surgery when HUD was used.

TABLE 4.

Utility of HUD by Pathology/Surgical Procedure

Pathology/surgical procedure	Measure of utility
Intra-axial/superficial lesions	Skin incision
	Craniotomy
	Dural opening
	Corticectomy
Skull base lesions	Head/bed positioning
	Extra and intradural drilling
Vascular lesions	Arachnoid opening
Transphenoidal approach	Head/bed positioning
	Extradural drilling
	Dural opening

FIGURE 3.

Bed/head positioning. A, A patient with a third ventricular hemangioblastoma. An interhemispheric and subfrontal approach was planned. HUD was activated prior to surgery. B, The patient's head was positioned with too much flexion, at first, and the HUD demonstrated that the approach to the lesion would transgress the frontal lobe. C, By extending the patient's head, the lesion was translated to a corridor along the skull base.

FIGURE 12.

Identification of normal appearing pathological tissue. A, A patient with a first-time grand mal seizure was found to have a nonenhancing, flair positive left frontal lesion as seen in this sagittal flair MRI. In this case, the flair positive area was painted (green). B, After dural opening, normal appearing cortex was encountered. C, The HUD was turned on and outlined the flair positive lesion. D, The resection of normal-appearing tissue was guided by the HUD.

Skin incision. A, A patient with diabetes presenting with hemiplegia had a MRI with diffusion weighted imaging sequences performed, which demonstrated an extensive frontoparietal abscess. B, The HUD was used to outline the abscess (yellow) and was activated early to help tailor the correct skin incision. C, The HUD was then projected onto the cortical surface to understand the abscess location beneath the cortical surface.

Craniotomy/craniectomy. A, A patient with a cerebellar meningioma at the junction of the transverse and sigmoid sinuses. B, The HUD was activated prior to performing the craniectomy and was used to demonstrate the tumor (green), dural sinuses (purple), and guide the craniectomy.

Extradural drilling. A, A patient with ataxia and hemiparesis was found to have large anterior foramen magnum meningioma, as seen here in this noncontrast sagittal MRI. The patient underwent a left far lateral craniectomy. B, The HUD was used to outline the tumor (yellow), vertebral artery (red), and brainstem (green). A C1 laminectomy was performed to reach the bottom of the tumor and drilling of the occipital condyle was tailored to reach the lateral aspect of the tumor as well as the intracranial vertebral artery.

Dural opening. A, A patient with headaches was found to have a large right-sided tentorial meningioma, as seen here in a contrast enhanced coronal MRI. B, The patient underwent a temporal craniotomy. The HUD was used to outline the tumor (yellow), and was used to help guide the dural opening.

Arachnoid opening. A, A patient with previous subarachnoid hemorrhage from a ruptured posterior inferior cerebellar artery aneurysm with aneurysm recurrence following coiling, as seen here on a lateral digital subtraction angiography. B and C, In this case, the aneurysm was painted (green). The HUD was used to tailor a focused arachnoid opening directly over the aneurysm.

Corticectomy. A, A patient with a lateral ventricular AVM that had previously undergone radiation, but was not obliterated, had undergone cystic change as seen in this contrast-enhanced coronal MRI with a planned temporal trajectory. In this case, the AVM was painted (red). The HUD was activated after dural opening and was used to choose a precise temporal cortisectomy (B) to reach this deep lesion (C). The HUD allowed for visualization of an accurate, narrow, and safe trajectory to a deep location.

Identification of nerves/vessels within tumor. A, In the same patient shown in Figure 10, the optic nerve is first embedded within the tumor and difficult to visualize. B, The HUD provides guidance as to its location, and once a portion of the tumor has been removed, the optic nerve is better visualized.

FIGURE 10.

Intradural drilling. A, A patient with a tuberculum meningioma with a lateral extent, as seen on contrast enhanced coronal MRI. Given the lateral extent, the patient was selected for a transcranial approach, specifically a bifrontal craniectomy and subfrontal approach to the tumor. In this case, the tumor (yellow), optic nerves (green), and carotid arteries (red) were painted. The optic nerve can be seen entering the optic canal and then taking a normal slightly lateral trajectory. B, The HUD is used here to understand the course of the optic nerve within the optic canal while drilling the orbital roof.

DISCUSSION

Main Findings

This series is, to our knowledge, the largest experience using HUD to assist with intracranial surgery. We have shown here that HUD can be used for a wide variety of both vascular and oncologic pathologies both on the surface and in the depths of the brain. Excellent or good accuracy was maintained in the majority of cases (91.6%). Deep lesions were less likely to have excellent accuracy. The overlay can be used to outline not only the pathological lesion, but also surrounding vessels and nervous tissue that must be anticipated, identified, and preserved.

We have also shown that HUD has potential value during multiple stages of surgery (other than the lesion localization/resection) from as early as the skin incision/positioning to arachnoid dissection and intracranial drilling. Our experience was that HUD utility varied depending on pathology. We found that for intra-axial and superficial lesions, HUD was more useful for skin incision, craniotomy, dural opening, and corticectomy. On the other hand, for skull base lesions, HUD was more useful for bed/head positioning as well as extradural/intradural bone removal. These findings are intuitive, as intra-axial lesions require more unique operative plans, whereas skull base lesions generally follow a more typical surgical approach and depend on bone removal for adequate lesion exposure. HUD used during bed positioning, skin incision, craniotomy, and dural opening represents a deviation from normal microscope workflow. We have demonstrated here that HUD can be potentially used during these phases, but only if the operator deems that it would be useful.

During the tranphenoidal approach, HUD can be useful for choosing the correct trajectory to the sellar region, defining the carotid arteries and optic nerves from the nasal cavity, and in turn guiding the craniectomy, dural opening, and tumor resection. Although navigation is typically not utilized during aneurysm and intracranial stenosis surgery, HUD proved to have some utility in these cases. During aneurysm surgery, HUD can be used to visualize the aneurysm and tailor the arachnoid dissection. In addition, we used HUD to identify the target recipient vessel during the bypass surgery. Finally, based on our small experience with low-grade gliomas, we postulate that HUD can be useful for guiding the resection of lesions that do not appear abnormal to the naked eye.

HUD Limitations

HUD itself has 2 major limitations (Figure 13). First, HUD relies on navigation accuracy and any loss of accuracy can potentially lead to false reliance on HUD. HUD accuracy can be affected by poor intraoperative navigation, inaccurate object painting, brain shift, brain retraction, and may also deteriorate as the surgery progresses. Navigation accuracy should be confirmed throughout each procedure to assure validity of information provided by the HUD. This involves vigilance from members of the operating team including surgeons, circulating and scrub staff, anesthesia, and intraoperative neurophysiology monitoring teams to assure that the navigation star with fiducials is not moved during preparation or other phases of the operation. Further, loss of accuracy has different implications for different pathologies. For instance, imperfect accuracy can be tolerated for lesions such as extra-axial tumors or aneurysms, where HUD may serve as a guide to the general vicinity of the lesion, but the lesion itself is obvious thereafter. On the other hand, excellent HUD accuracy is essential for normal-appearing lesions, such as low-grade gliomas, and deep intra-axial lesions that have no other landmarks. It is essential that the operator understands the importance of accuracy for each case.

Figure 13.

HUD limitations. The main limitations of HUD are inaccuracy as seen here with the HUD not accurately overlaying a cerebellar cavernous malformation (A) and distractibility as seen here during resection of a cerebellopontine angle AVM (B).

The second major HUD limitation is that the painted objects that are injected into the microscope can become distracting from the normal anatomy. There is a learning curve for visualizing normal anatomy while the HUD is active and for integrating information provided by the HUD graphical overlay. Distraction was cited as the reason for disabling the HUD in 38.7% of cases for which HUD was disabled during a case. There is certainly room for improvement in terms of seamlessly integrating HUD without disruption.

Previous Literature

There have been very few reports of HUD use in surgery. In 2016, Yoon et al[1] reported the use of HUD for placement of 40 pedicle screws. The authors used Medtronic StealthStation (Medtronic, Dublin, Republic of Ireland) with the HUD injected into Google Glass (Google, Mountain View, California) in this instance. They found the HUD to be useful, especially when the neuronavigation monitor was placed behind the surgeon so that the surgeon did not have to turn his or her head 180°. The surgeons felt they could maintain focus on the operative task without having to move their head or shift focus. HUD has been used in ophthalmologic procedures,[2,3] diabetic limb salvage surgery,[4] orthopedic procedures,[5] and bedside procedures such as central line placements.[6] Anesthesiologists have used HUD to view vital signs.[7] Many of these approaches utilize Google Glass. HUD has been used in the aviation industry for many years to project data on to the window in front of the pilots’ eyes. Similarly, automated surgical trajectories utilizing microscope-navigation integration have been described.[8]

The concept of augmented reality (AR) in neurosurgery has been explored as early as the 1990s,[9,10] including reports of injecting overlays into the operative microscope.[11] AR has been used in endoscopic transsphenoidal surgery with virtual images of the tumor and nearby structures overlaid onto the endoscopic tower view.[12] Kockro et al[13] described the Dex-Ray system in which a handheld probe on the skin surface integrated with projections on an adjacent screen. Deng et al[14] described an easy-to-use AR neuronavigation system using a tablet PC to view the virtual image. Cabrilo et al[15] described AR use for 28 patients with 39 unruptured aneurysms in which preoperative imaging was injected into the operative microscope. In this work, Brainlab was integrated with Zeiss, as it was in our study. The authors showed examples of bony anatomy projected onto the skin to tailor the incision, vessel anatomy projected onto the bony surface to tailor a craniotomy, as well as aneurysm projection onto the arachnoid to tailor the final dissection. The authors also describe its utility in positioning the head (10%), tailoring the craniotomy (63.3%), minimizing arachnoid dissection (66.7%), choosing clip position (92.3%), and its overall major impact (16.7%). The same authors also describe the use of AR in the treatment of AVMs and during bypass surgery.[16,17] They found it to be less useful for obtaining relevant information regarding feeding arteries during the AVM surgery, but helpful in identifying donor and recipient vessels during the bypass surgery, especially outlining the superficial temporal artery on the skin beforehand.

Limitations

Our study is limited first by its retrospective nature. A prospective assessment of accuracy and utility would improve the strength of the study. Secondly, our assessments of accuracy and utility are entirely subjective and therefore it is difficult to truly quantify the accuracy and utility of HUD. Our assessment of accuracy was subjective (not objective). Further, we only assessed accuracy once during a given case, rather than at multiple time points to demonstrate if there is accuracy deterioration. Finally, we did not record the source of lost accuracy (eg, brain shift vs poor registration vs poor painting), which is an important factor to understand. Although we reported utility by describing the phases of surgery in which HUD was used, the decision to use HUD in a given phase of surgery was entirely operator dependent. Finally, we have not demonstrated its use in comparison to non-HUD cases and we have not demonstrated an impact on outcome. In future investigations, it would be useful to assess operative time, surgical approach, extent of resection, and patient outcome in HUD and non-HUD cases.

CONCLUSION

Our early experience with HUD technology demonstrates that it can be safely used for a wide variety of vascular and oncologic intracranial pathologies and has potential value during multiple stages of surgery. A prospective assessment of the technology with predetermined endpoints is needed.

Disclosure

The authors have no personal, financial, or institutional interest in any of the drugs, materials, or devices described in this article.

Click here for additional data file.