Augmenting Microsoft's HoloLens with vuforia tracking for neuronavigation.

Major hurdles for Microsoft's HoloLens as a tool in medicine have been accessing tracking data, as well as a relatively high-localisation error of the displayed information; cumulatively resulting in its limited use and minimal quantification. The following work investigates the augmentation of HoloLens with the proprietary image processing SDK Vuforia, allowing integration of data from its front-facing RGB camera to provide more spatially stable holograms for neuronavigational use. Continuous camera tracking was able to maintain hologram registration with a mean perceived drift of 1.41 mm, as well as a mean sub 2-mm surface point localisation accuracy of 53%, all while allowing the researcher to walk about a test area. This represents a 68% improvement for the later and a 34% improvement for the former compared with a typical HoloLens deployment used as a control. Both represent a significant improvement on hologram stability given the current state-of-the-art, and to the best of the authors knowledge are the first example of quantified measurements when augmenting hologram stability using data from the RGB sensor.

Introduction

Neurosurgery, a field heavily reliant on medical imaging for preoperative planning and perioperative navigation provides an excellent, though challenging, landscape for the exploration of augmented realty in medicine. Numerous authors have described the disadvantages of current neuronavigation paradigms [1-8]. Of paramount concern is that the attention of the physician is divided between the procedural workspace of the patient and the navigational workspace of the displays on which the data are shown. This requires one to mentally transform and relate the visualised two-dimensional (2D) imaging data into 3D physical space, often without the ability to see both at the same time. Augmented reality (AR), using a head mounted device (HMD), addresses both problems by directly rendering 3D models of anatomy, planning information, or other pertinent data into the physician's field of view (FOV), aligned with the patient anatomy.

Research into the use of AR for neuronavigation has predominately focused on augmenting information displayed on an external screen, while initial registration and tracking have commonly been addressed using an outside-in approach, i.e. with the aid of external tracking hardware similar to existing commercial navigation systems. This trend is highlighted in the review paper from Guha et al. [1], where a majority of the 33 studies highlighted made use of some form of external tracking, whether for a hand-held camera coupled to an external monitor, or tablet-like device. In contrast, only four studies made use of an HMD, and only one of these was capable of self-pose estimation. This inside-out approach to tracking eliminates the need for external tracking hardware, reducing OR clutter, avoids line-of-sight problems and could prove to be considerably more cost-effective.

Microsoft's HoloLens, when released in 2016, was one of the first headset devices to implement AR on a commercial scale and has provided the biomedical field with a unified platform for the development of AR in medicine. Since then, it has been a focus of discussion and research in the context of neuronavigation, as it allows for the potential marriage of the patient with their medical imaging data; a long overdue paradigm shift in the art. It contains an inertial measurement unit, one front facing depth camera and four flanking grey-scale cameras used to map the spatial surrounding. Proprietary algorithms perform a fusion of the sensor readings to achieve simultaneous localisation and mapping (SLAM), aimed at providing a robust approximation of scene and pose. As such, it is possible to display computer models in the HoloLens' optical system, (‘holograms’ only insofar as Microsoft doublethink), and keep these holograms fixed, up to a certain accuracy, with respect to the physical space as the wearer changes their viewpoint and position.

The spatial alignment of data obtained from preoperative imaging to the patient anatomy is an essential component to any navigation system. The process is typically composed of two steps. During an initial registration step, the correspondence between the patient anatomy and computer model needs to be established. Next, that correspondence needs to be maintained, potentially even after the patient anatomy has moved, which we will refer to as tracking.

Research on the use of the HoloLens for medical interventions has suffered from the lack of access to the raw sensor readings of infrared (IR) and grey-scale cameras. This has resulted in the use of manual registration methods based on direct visual surface matching in lieu of more traditional automatic IR-based schemes [9, 10]. Furthermore, the perceived spatial stability of any placed hologram can only be as good as the headset localisation, and the inability to tune headset tracking for the close range, high-accuracy setting of surgery has resulted in tracking errors in the neighbourhood of ±6 mm [11-13].

Registration and tracking using HoloLens' contrived spatial mesh and SLAM system, without access to exact sensor information by Xie et al. [14] indicated that the low-vertex density and surface bias of the mesh, as well as the uncertainty of the SLAM, make this approach unsuitable. Garon et al. [15] went as far as to attach an external depth camera to a HoloLens, thereby bypassing the data limitation. However, neither of them reported on accuracy.

The notion of using the HoloLens' RGB camera and image processing techniques for tracking has been suggested as a workaround, as this is an already established technique in machine vision research. Towards the tail-end of papers, the proprietary image processing SDK Vuforia has often highlighted a solution in this regard, as it contains object tracking libraries compatible with HoloLens development [9, 16, 17].

Vuforia's image processing library allows HoloLens to track a known target image from the front facing camera's coordinate system. Therefore the apparent world position of any object whose position is made dependent on that tracked target is then independent of headset localisation. Not only does this approach provide the potential for improved hologram stability, but also for future automatic registration schemes.

Researchers from Duke University appeared to have used Vuforia for the tracking during a simulated external ventricular drain placement, however, no results or conclusion have been reported [18]. The use of RGB data from the front-facing camera in image processing has also been reported on by McDuff et al. [19] and Eckert et al. [20] who applied it towards ballistocardiography and object detection, respectively. Yet, to the best of our knowledge, no group has yet published quantitatively the advantages of augmenting HoloLens with object tracking information through the use of Vuforia, or other similar means.

In contrast to the media front by Microsoft concerning HoloLens as a tool within the medical field, the current body of published literature concerning its use in medicine is severely under quantified, with many publications (and media demonstrations) being of anecdotal quality. The following text expands on the current body of literature by quantifying the use of Vuforia's SDK in a simulated neuronavigational workflow.

Methods

Segmentation and 3D modelling

Imaging data were provided by physicians from UZ Brussels. Ultimately a 3D printed skull phantom, for which computed tomography data was available, was chosen. Obtained digital imaging and communications in medicine (DICOM) images were segmented using 3D Slicer to provide masks for bone and neoplastic structures. Segmentation was accomplished by a mixture of thresholding and cut-grow techniques using the plug-in Fast Cut Grow. In total three segmentations were made; the skull, and two mass phantoms (Fig. 1).

Fig. 1

Overview of the workflow from DICOM to the hologram

Following segmentation, 3D slicer was used to construct computer models of each structure. Model geometry was then smoothed using a Laplacian filter over 25 iterations and then exported in STL format. Due to the rendering hardware limitations of the HoloLens, the models were subsampled significantly using the open source program MeshLab. Subsampling was accomplished using a quadric edge collapse decimation filter. The complete computer model, as well as 3D printed phantom, was named Sara (Fig. 2).

Fig. 2

Sara model. Top row corresponds to (left to right) front, isometric, and right side view. Bottom row contains (left to right) top, bottom, and right side view

AR neuronavigation application

The personal distribution of the game development engine Unity (version 2017.2.0f3) was used to construct the main application which was to run on HoloLens. The decision to use an outdated Unity build, rather than the current 2018.1 build was due to known compatibility issues with the latest release (2017) of Microsoft's Mixed Reality Toolkit SDK (MRTK).

All holographic content was placed in a single parent object, whose persistent spatial anchor could be repositioned by the researcher. The models were not given mass or collision boundaries, as this would make it impossible to place them inside a phantom target or person. A single overhead directional light was used in the scene. In addition to this, the detailed phantom models were illuminated with global ambient lighting, ensuring a realistic interventional hardware load. The application was deployed to HoloLens after which it was ensured that performance never fell below 59 frames per second.

Object tracking

The Vuforia SDK (Version 6.5.22) was used for target tracking. To attenuate fluctuations in Vuforia's tracking output, the positional and rotational averages of 16 frames were taken using a sliding window; no appreciable delay or ill-effect was observed. During tracking, the model's transform was updated as the product of the camera–target transform and target–phantom transform.

Phantom tracking was accomplished through Vuforia's proprietary feature detection algorithms and a known RGB cylindrical target. Through comparison of extracted visual features on the tracking target and those of the computer model in memory, the 3D rigid transformation of the target can be estimated using a single camera. A priori knowledge of a transform between a well manually registered hologram and the phantom allowed for the consistent automatic manual registration of the hologram to its phantom target when using Vuforia. Continual tracking of the target allowed for this perceived registration to be maintained from any perspective.

Early on, plane image targets were found to be impracticable due to the line of sight limitations, and reduced tracking accuracy at high incidence angles between the target and camera. The implementation of a cylindrical target solved this by allowing the target to remain normal to the headset's camera over 360°. To that end, the well performing provided stock Vuforia image tarmac was printed onto semi-mat photo paper at 1200 dots per inch. Measuring 165.7 mm in height and 235.6 mm in width, it was then wrapped around a cylinder whose diameter measured 75 mm and secured by thin double sided tape in such a manner where it would not be visible (Fig. 3b).

Fig. 3

Experimental setup

a Illustration of measurement points relative to the phantom

b View perspectives of the phantom from each angle of measurement

Phantom setup

To measure hologram accuracy, an A4 sheet of millimetre paper whose minor lines were of 2 mm spacing, and whose major lines were of 10 mm spacing was affixed to a coated expanded polystyrene support board. To this was then affixed through a non-permanent bonding agent the 3D printed Sara skull phantom. This assembly was then secured using common camera mounting hardware to the working end of a tripod. The pitch, yaw, and a roll of the tripod's head, and by extension the phantom, were confirmed to be <1°. The distance between the floor and the XY plane of the millimetre paper was 130 cm.

To assure that the obtained data were comparable, fixed locations were chosen for repeatability. On the floor, marked points were placed at −90°, −45°, 0°, +45°, and +90° at the distances of 40 and 80 cm from the centre of the phantom, Fig. 3a. These measurement positions were chosen to replicate the distance and range of movement a surgeon may experience during the planning phase of an intervention.

The phantom had been scanned and modelled with a series of radio-opaque fiducials placed at various surface points (Fig. 2). These were removed from the physical model, however, their exact placement was marked on the phantom itself.

Experimental conditions

Two test conditions were designed to test the efficacy of augmenting HoloLens with Vuforia. The control condition represents a typical device configuration, while the Vuforia test condition included target tracking. As the test lab is static, there was no need to update the spatial map during measurements. A detailed spatial mesh was made of the lab and saved to memory for retrieval.

Experimental metrics

Registration time. Quantified by recording the time required by the researcher to align the hologram with the phantom using programmed translational and rotational hand gestures. The perceived alignment of surface features was used a criterion for proper registration.

Hologram drift. Quantified from the perceived translation of the hologram relative to the phantom in the coordinate system of the underlying millimetre paper. Translation in the millimetre paper's Z direction was measured using a ruler (Fig. 4b).

Fig. 4

Example of measurement techniques

a Measuring localisation accuracy by placing the tip of the stylus into the centre of the holographic fiducial

b Measuring perceived holographic drift by the difference in similar points. Note: The apparent misalignment between the phantom and the hologram seen in both figures is due to the displacement between the RGB camera used to record the scene and the wearer's line of sight. From the wearer's perspective, the hologram is where it should be

Localisation accuracy. Quantified by the ability of the researcher to locate a surface point on the phantom based on holographic markers. The centre fiducial points of the three front markers, still visible on the hologram but not the phantom, are identified by a researcher using a fine tipped stylus. The radial distance from the known true centre and the perceived centre is recorded (Fig. 4a).

Experimental methodology

Registration was performed once at the beginning of each trial with the researcher standing at 0° and 80 cm distance; this was either manual for the control condition or automatic in Vuforia condition. Only for control trials was registration time noted, as the automatic method was near instantaneous. From this position, both perceived hologram translation and surface point localisation accuracy were noted. The researcher then moved up to the 40 cm distance and repeated the measurements. Following this pattern of 80–40 cm distances, data was collected, in order, from the −45°, −90°, +45°, and +90° positions. Measurement data obtained from both distances were then aggregated to a single angle.

Multiple iterations of each test condition were performed by an engineering researcher intimately familiar with the program and controls, resetting the program between use. By iterating each test condition it was possible to smooth variance in data, and maintaining a single researcher helped to ensure a consistent bias. This is important as the exact registration of holograms manually is never guaranteed, and translational drift errors rely on the subjective opinion of the researcher. It should be noted that this subjectiveness results from a difficulty to accurately judge matter-hologram boundaries, discussed more in depth later. The decision to reset the program between trial runs was to assure a cold-start, clearing out any data which may have effected future results, and better simulating a clinical use case. Cumulatively data was collected and averaged from 19 trials into the two test conditions.

Results

Manual registration times yielded a mean time of 95 s, with a minimum and maximum time of 51 and 165 s, respectively, summarised in Table 1.

Table 1

Comparison of published manual hologram registration times

Study	Range, s	Mean, s
Proposed method	51–165	95
Pratt et al. [10]	60–120	NR
Rae et al. [9]	135–219	172

NR: not reported.

The mean perceived holographic drift of the control condition was 4.39 mm. The Vuforia test condition yielded 68% improved results, with a mean perceived holographic drift of 1.41 mm; summarised in Table 2. The reduction in perceived drift was statistically significant at all measurement angles, except for 0°, per Student's t-tests at 5% confidence.

Table 2

Change in mean perceived drift for each measurement angle

Condition	−90°	−45°	0°	+45°	+90°	Mean	σ	SEM
control, mm	6.27	3.59	0.62	4.60	6.90	4.39	3.34	1.29
vuforia, mm	0.83	1.46	1.24	0.08	3.42	1.41	1.08	0.67
Δ	−87%	−59%	0%a	−98%	−50%	−68%	−68%	−48%

aShown to be insignificant.

, standard deviation.

SEM, standard error of the mean.

The standard error of the mean (SEM) among all trials for the control condition was 1.29 mm. The Vuforia test condition yield 48% more consistent results with a SEM of 0.67 mm.

Surface point localisation in the control condition showed a mean error of 5.43 mm. The Vuforia test condition yields a 65% reduction, with a mean error or 1.92 mm. This resulted in a 34% improvement in sub 2-mm accuracy versus the control; summarised in Table 3. All three points showed a statistically significant improvement versus the control per Welch's t-test at 5% confidence.

Table 3

Comparison of surface localisation results

Study	<2 mm	2–5 mm	5–10 mm	>10 mm
Rae et al. [9]	50%	0%	50%	0%
Proposed method
Matcheda	87%	13%	0%	0%
Control	19%	34%	40%	7%
Vuforia	53%	40%	7%	0%

aResults matched to reflect measurements taken from a starting position, without introducing movement.

Discussion

Registration time

The obtained data highlights the extreme variability in the ability of the user to register, as best they may, a hologram based on surface features alone. This observation is consistent with the ranges of registration time reported in the literature, summarised in Table 1. When comparing the obtained manual registration times with those of Rae et al. [9], when matched like-for-like for an experienced user, our results represent a 45% reduction in mean time and a 62 and 25% reduction in lower and upper results, respectively.

Pratt et al. [10] reported times between 1 and 2 min per registration, the upper result showing a 27% advantage over our results. It should be noted though that unlike this study and that of Rae et al., registration was performed on human legs intraoperatively, rather than on head phantoms in a lab.

Cumulatively, registration times from the reporting papers in the Guha et al. [1] review of AR, ranged from 180 to 960 s, with a mean of 429 s. However, due to the variety in AR implementation and registration techniques between published papers, it would not be appropriate to make direct comparisons between these results and the obtained data. What is clear though is that the use of HoloLens as an AR platform appears to be of great benefit in this metric.

The difference in registration times can be attributed to a number of factors, particularly user experience and control schemes. Dissatisfied by those provided in the MRTK, and taking inspiration from the model controls inside of Unity, the iterative collaboration between the engineering team and neurosurgical staff resulted in controls which allowed for precise per-axis model transformation from any perspective; outlined in Fig. 5.

Fig. 5

Control schemes used for manual transforms of models

a Translation: allows the user to axially move the model in either the model or world coordinate system using the ‘pinch and drag’ command

b Rotation: Allows the user to highlight any axis of the model coordinate system with their gaze and ‘pinch and drag’ to rotate along with it

Perceived drift

Table 2 summarises the difference in mean perceived drift between experimental conditions. The lack of any statistical improvement at 0° is not surprising as the hologram would have been registered either manually or automatically from this position, and the visual effects of the holographic drift are often most prevalent while walking around a hologram, rather than towards it.

Predominately, the perceived drift when using Vuforia tracking was under 2 mm for all angles except for +90°, where at 3.42 mm it was 278% greater than the mean of other positions. This anomaly was consistent across all trial runs, suggesting that the tracking target may have been compromised in some capacity from that measurement perspective. Despite this, augmenting HoloLens with Vuforia yielded a mean decrease in the perceived drift of 68% than without.

This reduced SEM across Vuforia trial runs is most likely due to the consistent transform used for registration. In contrast to this, the less consistent results from the control are most likely due to the difficulty in perceiving exact hologram–matter interactions during manual registration, resulting in poor depth judgment between the two. This is most pronounced when a hologram is registered from a single perspective. Fig. 6 highlights this effect.

Fig. 6

Perception of hologram–phantom relationship from a single view

a Similarity of an observed hologram relative to its phantom from a single perspective

b Two manual registrations highlighting manual registration error. Top row: view from standing; bottom row: view from head on. Variance in vertical height between each registration may be seen

Auvinet et al. [11] reported deviations in headset localisation of ±5.6, ±4.4, and ±5.2 mm along each cardinal direction. Liu et al. [13] reported average headset deviations of 5.6, 20.6, and 133.8 mm for slow, quick, and rapid head movements, respectively. Vassallo et al.[12] reported mean hologram localisation accuracy of 5.83 ± 0.51 mm while trying to disrupt its tracking through various means. These are all comparable to the perceived drift obtained in the control condition.

Point localisation

Rae et al. reported on similar metrics used in their experiments, and a comparison can be found in Table 3. It is worth noting that due to the known holographic drift, their group measured results from only a single position after performing manual registration of the hologram. In consideration of this, we have included a selection of matched results which were taken following registration from the 0° position. It can be seen that before introducing movement associated holographic drift errors, there is a marked improvement versus literature.

This trend continued when analysing all the collected data. Implementing Vuforia tracking yielded similar or better results accuracy across all threshold categories when compared with the published literature, while still allowing unabated movements within the test parameters.

Remarks

One of the challenges when using the active RGB camera tracking was the absolute need to maintain a line of sight between the target and the phantom. This, while not prohibitively difficult at greater distances from the phantom did become so at an arm's length distance, where the target, though within visual line of sight, was outside the frustum of the camera. This may be remedied by better placement of the tracking target, use of multiple targets at various key points, and/or rethinking its form. For example, using the RGB camera to track 3D printed objects rather than RGB targets, a feature Vuforia already supports. As for the practicality of such a large marker, Vuforia, or alternative image processing algorithm(s), may effectively be used to maintain an established registration through a secondary reference, the initial registration possibly obtained through the tracking of a smaller stylus and surface matching. This is indeed how workflows are often structured in commercial markerless neuronavigation systems; however, unlike these bulkier systems, HoloLens could easily be deployed beyond the walls of an OR.

Limiting manual registration to a single perspective was arguable to the detriment of accuracy in the control, however, it does highlight the necessity to adopt automatic registration into AR platforms. This is particularly true in situations where the wearer has neither the space nor time to manually adapt the registration from numerous perspectives, especially if the registration from one perspective is disturbed due to the tracking inaccuracies of the headset as he/she changes perspectives.

In the scope of medicine, the authors feel that the use of any spatial mapping information is superfluous, as its principal role is to allow holograms to interact with the environment; an unnecessary criterion when one wants to place a hologram inside a head rather than on top of it. A decision to not load, generate, or update any spatial meshing data may therefore further free up already limited hardware resources for future visualisation, tracking, or registration tasks.

With the recent release of Microsoft's beta Redstone firmware update for HoloLens, the prospect of using data collected from the front facing depth camera becomes reality. Not only does this camera have a greater FOV than the RGB camera, 120° versus 48°, but it also allows for tracking of more familiar, and already in place, instruments in the OR through IR reflectivity.

Having been released in 2016, HoloLens has begun to look outdated versus the competition. Over the last few years, the AR industry has not stood idly by, and recent HMD releases from ODG, Meta, and Magic Leap represent significant improvements over HoloLens. Of particular interest are those systems whose optics promise greater depth perception. Still, it is expected that Microsoft will release the replacement for HoloLens, codenamed Sydney, sometime in 2019, the delivery of which will keep the workflow and technologies used through this Letter relevant for years to come.

Conclusion

It was demonstrated that by augmenting HoloLens with Vuforia's RGB target recognition for inside out tracking, significantly greater hologram stability could be achieved than without; thus alleviating one of the greatest hurdles for HoloLens as a tool in neuronavigation. Furthermore, the greater variance between trials obtained when performing registration manually, even by those experienced with HoloLens, highlights the need for more effort placed on automatic workflows, and a re-examination of current practices among many professionals.

Funding and declaration of interests

None declared.