Patient-specific virtual reality technology for complex neurosurgical cases: illustrative cases.

BACKGROUND: Virtual reality (VR) offers an interactive environment for visualizing the intimate three-dimensional (3D) relationship between a patient's pathology and surrounding anatomy. The authors present a model for using personalized VR technology, applied across the neurosurgical treatment continuum from the initial consultation to preoperative surgical planning, then to intraoperative navigation, and finally to postoperative visits, for various tumor and vascular pathologies. OBSERVATIONS: Five adult patients undergoing procedures for spinal cord cavernoma, clinoidal meningioma, anaplastic oligodendroglioma, giant aneurysm, and arteriovenous malformation were included. For each case, 360-degree VR (360°VR) environments developed using Surgical Theater were used for patient consultation, preoperative planning, and/or intraoperative 3D navigation. The custom 360°VR model was rendered from the patient's preoperative imaging. For two cases, the plan changed after reviewing the patient's 360°VR model from one based on conventional Digital Imaging and Communications in Medicine imaging. LESSONS: Live 360° visualization with Surgical Theater in conjunction with surgical navigation helped validate the decisions made intraoperatively. The 360°VR models provided visualization to better understand the lesion's 3D anatomy, as well as to plan and execute the safest patient-specific approach, rather than a less detailed, more standardized one. In all cases, preoperative planning using the patient's 360°VR model had a significant impact on the surgical approach.

The continuing advancement of neuroimaging techniques is critical to refine and improve both surgical planning and predicting prognosis.[1] However, visualization and navigation challenges still exist for complex pathologies that are difficult to diagnose and/or are located in hard-to-access regions.[2] Over the last two decades, the adoption of virtual reality (VR) in medicine has been driven by technological advancements. In parallel, various applications, including medical training,[3-8] patient education,[9,10] therapeutic intervention,[11-13] and surgical intervention,[14-18] have been explored. VR combined with three-dimensional (3D) fusion imaging allows data from different modalities to be analyzed in the same space by providing immersive visualization that is reproducible and shareable.[5,7,15-17]

The Surgical Theater visualization platform provides a proven method of digitally rendering patient-specific 360-degree VR (360°VR) models that allow users to become immersed in the patient’s unique anatomy when integrated with a VR headset. The Surgical Rehearsal Platform (SRP; Surgical Theater) can import volumetric data from up to eight imaging modalities and fuse them in 3D. It can display accurate modeling of surgical tools within the 360° environment of the patient’s anatomy for preoperative planning and rehearsal. Multiple surgical trajectories and corridors can be simulated in order to plan an optimal approach. Intraoperatively, the platform’s Surgical Navigation Advanced Platform (SNAP; Surgical Theater) can connect to external surgical navigation systems to extract and display navigation data inside the model. SNAP can provide a view of the model from the navigation probe tip as it is moving within the patient’s brain, as well as give the surgeon a 360° view of the pathology. The software’s selective clipping and opacity control tools allow real-time manipulation of structures that are otherwise hidden. The use of Surgical Theater for various applications has been previously reported.[8,16,19-23]

Here, we report the application of the dynamic VR technology across the neurosurgical treatment continuum in a case series of complex tumor and vascular pathologies. The resulting impact of this technology on pathology management is discussed.

Illustrative Cases

Five patients undergoing procedures for various pathologies at the participating institutions were included (Table 1). Institutional review board approval was not necessary, as the patients’ clinical course or standard of care did not change.

TABLE 1.

Pathology, imaging for preoperative 360°VR model, and Surgical Theater use for each case

Case No.	Age (yrs), Sex	Pathology	Imaging for Preop 360°VR Modeling	Surgical Theater Use
				Preop Consult & Postop Visit	Preop Planning	Intraop 3D Navigation
1	41, M	Rt anterior clinoidal meningioma	3D MP-RAGE MRI postcontrast; CTA w/ & w/o contrast	X	X	X
2	62, M	Lt parietal anaplastic oligodendroglioma	T1-weighted SPGR MRI w/ & w/o contrast; DTI tractography; axial motor fMRI; CTA isotropic	X	X	X
3	29, M	Intradural intramedullary spinal cord cavernoma	Axial CT; axial 3D MP-RAGE MRI; sagittal T2-weighted MRI; axial DTI tractography	X	X	X
4	22, M	Giant ruptured rt M1 MCA aneurysm	CTA w/ contrast		X	X
5	27, F	Lt posterior parieto-occipital ruptured AVM	Axial BRAVO stereotactic MRI w/o contrast; DTI tractography	X	X	X
Supplementary case 1	65, F	Rt sylvian ruptured AVM	Axial BRAVO stereotactic MRI w/ contrast; sagittal T2-weighted CUBE MRI w/ contrast; T1-weighted fMRI w/o contrast; 3D DSA; DTI tractography	X	X	X

BRAVO = brain volume; MP-RAGE = magnetization-prepared rapid gradient echo; SPGR = spoiled gradient recalled.

Preoperative Planning With 360°VR

A single interactive 360°VR model was created from each patient’s volumetric scans in Digital Imaging and Communications in Medicine (DICOM) format using the SRP. The software digitally stacked each scan’s layers in space, accurate to the slice thickness. Diffusion tensor imaging (DTI) was post-processed by setting anatomically defined regions of interest. The model was thresholded to show individual structures within specified intensity ranges. Multiple imaging modalities were incorporated to visualize different structures (Table 1).

The 360°VR models were used by the surgical team at each institution to plan, review, and/or rehearse the procedure using VR headsets on the SRP outside of the operating room (OR) and/or on SNAP in the OR before scrubbing in. One or more trajectories were planned and reviewed.

360°VR Consultation

Patients were shown both their preoperative and postoperative 360°VR models. They manipulated the virtual anatomy models of themselves via touchscreen controls or VR controllers. The risks and benefits of the proposed procedures and treatment alternatives were discussed.

Intraoperative Navigation With 360°VR

SNAP containing the patient’s 360°VR model was integrated with external neuronavigation systems, allowing live 3D tracking and visualization. SNAP provided views of the surgeon’s choosing and tracked the probe’s position and orientation and adjusted the model accordingly. Upon the surgeon’s request, a clinical specialist manipulated the SNAP scene.

Case 1: Clinoidal Meningioma

A 41-year-old male presented with mild right-sided headaches and blurred vision in his right eye. Magnetic resonance imaging (MRI) revealed a 1.4 × 1.5 × 1.6–cm uniformly enhancing extra-axial lesion wrapped around the right anterior clinoid, extending along the lateral margin of the right optic nerve as it exited the optic canal and posterior to the optic chiasm (Fig. 1). A supraorbital eyebrow keyhole craniotomy was initially recommended on the basis of DICOM. On the morning of surgery, viewing the patient’s 360°VR model from the back of the orbit revealed an inferior lobular extension of the lesion behind the superior orbital fissure (Fig. 1D–F). A “fly-through” review of the simulated craniotomy corridor and virtual drilling showed this approach would yield an incomplete resection. The approach was modified to a minipterional craniotomy, which was quickly planned and consulted with the patient in 360°VR (Fig. 1G–I).

FIG. 1.

DICOM and 360°VR snapshots from case 1. A–C: Coronal, axial, and sagittal MRI views of the patient’s right anterior clinoidal meningioma in DICOM format. D: Snapshot of the patient’s 360°VR model of the clinoidal meningioma (green) in relation to the optic nerve (blue). E and F: Snapshots of the 360°VR model showing the surgical corridor and close-up views of the tumor at various angles for the supraorbital eyebrow keyhole craniotomy (original approach). G–I: Snapshots of the patient’s 360°VR model showing the surgical corridor and close-up views of the tumor at two different angles for the right minipterional craniotomy (revised approach).

The right minipterional craniotomy was performed. After drilling down the sphenoid wing, both the superior aspect of the tumor extending up along the anterior cranial fossa and the inferior aspect hidden behind the superior orbital fissure were apparent. A microscope was used along with neuronavigation integrated with SNAP for tumor visualization, circumferential microdissection, and internal tumor debulking. The tumor was resected completely without complication. Postoperative day 1 MRI demonstrated gross-total resection. The patient’s vision immediately recovered, and he was discharged after 2 days (Video 1, time stamp 0:00).

VIDEO 1. Clip showing cases 1–3. Click here to view.

Case 2: Anaplastic Oligodendroglioma

A 62-year-old male presented with a transient episode of vertigo, right lower extremity weakness, and coordination difficulty on the right side. Computed tomography (CT) showed a calcified mass on the left parietal lobe abutting the interhemispheric falx. MRI and functional MRI (fMRI) revealed a 3.4 × 2.5 × 3.9–cm ring-enhancing lesion involving the medial left parietal lobe, located posteromedial to the left motor cortex and medial to the left corticospinal tract bundles (Fig. 2A–C). The initial plan based on DICOM was to approach ipsilaterally via a left parietal craniotomy. However, this was changed to a right parietal craniotomy for a transfalcine, interhemispheric approach after reviewing in 360°VR (Fig. 2D–F). Planning in 360°VR allowed identification and outlining of the vein of Trolard in relation to the lesion to explore the interhemispheric trajectory. The two approaches were simulated and compared using the 360°VR model with incorporated DTI and fMRI to better understand the necessary craniotomy angles and degrees of retraction (Fig. 2G and H). The transfalcine approach was verified to result in less retraction onto the ipsilateral hemisphere, reducing the risk of motor deficit.

FIG. 2.

DICOM and 360°VR snapshots from case 2. A–C: Axial, sagittal, and coronal MRI views of the patient’s left parietal anaplastic oligodendroglioma in DICOM format. D–F: Snapshots of the patient’s 360°VR model in corresponding views. G and H: Snapshots of the patient’s 360°VR model displaying the original ipsilateral approach via a left parietal craniotomy (G) and the chosen transfalcine approach via a right parietal craniotomy (H).

Neuronavigation was used with SNAP to guide the incision, confirm tumor borders, and identify the margins of the motor cortex and corticospinal tract bundles. The tumor was resected without complication, leaving a small linear region of tumor anteriorly that invaded the motor cortex, which was treated with Temodar (temozolomide, Merck). The patient awakened with his motor function intact (Video 1, time stamp 2:35).

Case 3: Spinal Cord Cavernoma

A 29-year-old male presented with pain localized to the interscapular region and numbness from the waist down. Spine MRI showed an expansile T2-hyperintense intramedullary lesion posterior to the T3–4 disc space with interval resolution of cord edema above and below (Fig. 3A–C). The tumor was densely adhered to surrounding nerve tissue. His 360°VR model confirmed the cavernoma’s location and resection approach with a T3–4 osteoplastic laminectomy and myelotomy (Fig. 3D–F). 360°VR was used to assess the cavernoma’s relationship to bony anatomy and surrounding critical fiber tracts, as indicated by DTI.

FIG. 3.

DICOM and 360°VR snapshots from cases 3 and 4. A–C: Preoperative imaging in DICOM format of the intramedullary spinal cord cavernoma in case 3. D–F: Snapshots of the 360°VR model of the cavernoma (red) reconstructed from the patient’s preoperative CT, MRI, and DTI. E and F: Close-up views of the cavernoma’s relationship to spinal column and DTI tracts. F: Snapshot showing surgical corridor to tumor with trajectory with semitransparent spine (right). G–I: CT/CTA/DSA views of the patient’s giant aneurysm and subarachnoid hemorrhage in case 4. J: Partially recanalized aneurysm. K–M: Preoperative snapshots of the 360°VR model of the giant aneurysm. N: 360°VR snapshot of retrograde aneurysm before the second surgery to trap the aneurysm.

In the OR, stereotactic registration for the intraoperative CT with spinal navigation and SNAP was performed. A decompressive T3–4 osteoplastic laminectomy was performed using standard protocols. Myelotomy was performed, and the tumor was isolated via dissection guided by microscope-integrated stereotactic navigation and SNAP visualization. The patient felt subjective improvement in the saddle region and right lower extremity on postoperative day 1. After 13 months, he was without any new neurological deficits. MRI revealed no evidence of residual or recurrent cavernoma (Video 1, time stamp 6:00).

Case 4: Giant Ruptured Middle Cerebral Artery Aneurysm

A 22-year-old male presented with a severe headache, nausea, and vomiting. CT/CT angiography (CTA) and digital subtraction angiography (DSA) confirmed a subarachnoid hemorrhage secondary to a 2.5-cm ruptured right M1 middle cerebral artery (MCA) aneurysm (Fig. 3G–J). His 360°VR model showed that clipping would not be optimal because of the M1 artery’s dysplastic nature and his young age (Fig. 3K–M). One inflow M1 artery and two outflow M2 arteries were clearly identified, and their relationship to the skull base was defined by the 360°VR model. An alternative approach to deliberately occlude the parent M1 vessel after performing a bypass from the superficial temporal artery (STA) to M4 of the MCA was decided after reviewing in 360°VR.

SNAP showing the 360°VR model and planned right-sided frontotemporal trajectory was integrated with neuronavigation. A frontotemporal craniotomy was performed, followed by an end-to-side anastomosis between the STA and M4 artery. The right M1 MCA was occluded with a clip. The patient had no changes in his neurophysiological monitoring parameters, and he awakened neurologically intact. However, he developed left hemiparesis that evening. MRI showed a small right MCA acute infarct, and DSA demonstrated complete thrombosis of the giant aneurysm but with some propagation of clot into the superior M2 branch causing a focal stenosis.

He recovered well over the next 3 months. DSA showed partial recanalization of his aneurysm because of retrograde filling from the STA–MCA bypass. A second surgery was required to completely trap the aneurysm. Surgical Theater was used for preoperative planning and intraoperative occlusion of the two M2 branches exiting the recurrent aneurysm because conventional imaging did not clearly define the exiting M2 branches (Fig. 3N). Surgery was completed without complications. Postoperative DSA demonstrated complete obliteration of the aneurysm with a widely patent bypass supplying his MCA distribution. He was doing well 2 years later, cognitively normal, and ambulating well with a left-hand paresis (Video 2, time stamp 0:00).

VIDEO 2. Clip showing cases 4 and 5 and supplementary case 1. Click here to view.

Case 5: Left Posterior Parieto-Occipital Ruptured Arteriovenous Malformation

A 27-year-old female presented during her third trimester of pregnancy with an acute hemorrhage from a 3-cm-deep, left parietooccipital Spetzler-Martin grade 4 arteriovenous malformation (AVM) causing a right homonymous hemianopsia. She had a cesarean section delivery and, after two sessions of endovascular embolization, underwent resection surgery. A 360°VR model rendered from her stereotactic MRI and fMRI clearly showed the spatial relationship of feeding branches from the left posterior cerebral artery, MCA, and draining veins, largely into the vein of Galen complex, all with respect to the optic radiations, and the corticospinal tract (Fig. 4A–E). Simultaneous visualization of these data sets within the model was critical in preoperative planning to avoid injury to these delicate pathways. The trajectory was planned to work through the encephalomalacia cavity from the prior bleed. The AVM was removed after circumferential dissection in the plane between it and surrounding hemosiderin-stained brain. Intraoperative DSA confirmed complete resection. Neuronavigation was used with the surgical microscope, both communicating with SNAP. All were crucial to AVM exposure and resection. Her 360°VR model was extensively referenced to ensure the least disruption of her already compromised optic radiations and preservation of the critical motor/sensory tracts. She tolerated the procedure well and remained neurologically normal, except for her baseline left homonymous hemianopsia. She was discharged 4 days postoperatively and was doing well 1 year later (Video 2, time stamp 3:50).

FIG. 4.

DICOM and 360°VR snapshots from case 5. A and B: Sagittal and axial MRI of the patient’s left posterior parieto-occipital AVM in DICOM format. C: Anteroposterior DSA view of the patient’s left posterior parieto-occipital AVM in DICOM format. D and E: Snapshots of the patient’s 360°VR model with incorporated DTI tractography showing the parieto-occipital AVM in relation to white matter fiber tractography at various angles.

An additional case describing a right sylvian ruptured AVM is presented in supplementary case 1 (Supplementary Fig. 1).

Discussion

Observations

Surgical Theater allowed planning the safest surgical approach by providing a detailed visualization of each patient’s unique anatomy. Reviewing the 360°VR model led to modifications of the initial approach in cases 1 and 2. Preoperative assessment of the clinoidal meningioma revealed the surgical corridor from the original craniotomy planned would not have provided a direct line of sight to the whole tumor. This change in the approach led to gross-total resection. For the oligodendroglioma, 360°VR planning allowed selection of the contralateral approach that avoided the vein of Trolard and did not require significant retraction of the ipsilateral hemisphere, lowering the risk of motor deficit. These cases support previous findings showing that planning with 3D simulation platforms including Surgical Theater can lead to modification of the surgical or treatment plan.[24] The comprehensive spatial information on the spinal cord cavernoma’s relationship to surrounding critical fiber tracts was useful for planning and resection.

The custom 360°VR models were especially useful in defining vascular pathologies and their 3D relationship to surrounding critical structures, eliminating the need for a mental construction and manipulation of multiple 2D images. The model played a crucial role in demonstrating the dysplastic MCA at the giant aneurysm’s base (case 4). The 360°VR model before the second surgery was essential in clearly delineating the location of the two exiting M2 branches. For the AVM, the malformation’s relationship with surrounding critical functional pathways was examined in detail at multiple angles with Surgical Theater. In supplementary case 1 (Supplementary Fig. 1), the availability of the interactive 3D representation of the different data sets in one space during the procedure was essential in performing a successful resection without damaging the surrounding eloquent cortex. In four cases, patients were consulted with custom models, which were useful for describing their pathology and simulating the recommended procedure in their unique anatomy. In case 1, simulating both approaches in 360°VR helped facilitate the discussion on the necessary change and alleviated anxiety. By providing interactive 3D visual information, consultations with patient-specific models have been proved to improve patient-reported understanding and satisfaction for various pathologies.[20,25]

Lessons

Advanced 3D planning in neurosurgery has been shown to facilitate the understanding of complex relationships between the lesion and surrounding anatomy.[26] For all cases, the use of patient-specific models for planning and intraoperative navigation improved the surgeons’ understanding of each patient’s unique anatomy and consequently their situational awareness while operating. The combined use of navigation with 360° visualization has proved beneficial in helping achieve maximal safe resection. Without this VR system, there would be no way to see and track surgical position within a 3D model of multiple scan types and functional data in the form of fMRI and DTI.

The immersive VR experience provides a novel way to educate patients and their families about their condition and management options. By allowing patients to learn and engage in a way not possible with conventional imaging, 360°VR consultations may ultimately result in true informed consent, which is often not met.[27-29] In addition to increased understanding and satisfaction, 94% of patients reported feeling included in the decision-making process during 360°VR consultations.[25] Patient involvement in shared decision-making may reflect a more comprehensive understanding of the treatment plan, which directly translates to more informed consent.[30-32]

Future work addressing the limitations of the qualitative findings of this paper might seek to measure quantitative information that supports the higher safety claims presented here. Such information might include operating time, percentage of undisturbed white matter fiber tracts or residual tumor postoperation, and/or number of visits before obtaining consent. Quantitative data that support this model of a patient-specific VR approach to surgery will drive department-wide changes in workflow that ultimately result in safer, more patient-engaged surgeries.

Acknowledgments

We gratefully acknowledge Daniela Barbery, Alex Yefimov, Keiji Drysdale, Reema Shah, and Malie Collins for case building of the 360°VR models and technical support. We also thank Christine Plant for assistance with the manuscript.

Disclosures

Ms. Anthony reported nonfinancial support from Surgical Theater during the conduct of the study. Dr. Louis reported personal fees from Surgical Theater and Medtronic outside the submitted work. Dr. Steinberg reported personal fees from Surgical Theater outside the submitted work.

Author Contributions

Conception and design: Steinberg, Louis, Steineke. Acquisition of data: all authors. Analysis and interpretation of data: Steinberg, Anthony, Louis, Shekhtman, Steineke. Drafting the article: Steinberg, Anthony, Louis, Shekhtman, Frempong-Boadu. Critically revising the article: Steinberg, Anthony, Louis, Shekhtman, Steineke. Reviewed submitted version of manuscript: all authors. Approved the final version of the manuscript on behalf of all authors: Steinberg. Administrative/technical/material support: Anthony. Study supervision: Steinberg.