Gesture-based registration correction using a mobile augmented reality image-guided neurosurgery system.

In image-guided neurosurgery, a registration between the patient and their pre-operative images and the tracking of surgical tools enables GPS-like guidance to the surgeon. However, factors such as brainshift, image distortion, and registration error cause the patient-to-image alignment accuracy to degrade throughout the surgical procedure no longer providing accurate guidance. The authors present a gesture-based method for manual registration correction to extend the usage of augmented reality (AR) neuronavigation systems. The authors' method, which makes use of the touchscreen capabilities of a tablet on which the AR navigation view is presented, enables surgeons to compensate for the effects of brainshift, misregistration, or tracking errors. They tested their system in a laboratory user study with ten subjects and found that they were able to achieve a median registration RMS error of 3.51 mm on landmarks around the craniotomy of interest. This is comparable to the level of accuracy attainable with previously proposed methods and currently available commercial systems while being simpler and quicker to use. The method could enable surgeons to quickly and easily compensate for most of the observed shift. Further advantages of their method include its ease of use, its small impact on the surgical workflow and its small-time requirement.

Introduction

In neurosurgery, surgeons treat different disorders which affect the brain, spinal cord, peripheral nerves, or cerebrovascular system. In order to do so, they first diagnose the disease and make surgical plans using preoperative images, such as magnetic resonance images (MRI) or computed tomography (CT). Having access to the spatial location and extent of a lesion (e.g. a tumour, arteriovenous malformation etc.) is crucial to the success of a surgical procedure. Providing surgeons with this type of information in the operating room have been one of the driving forces behind the development of image-guided surgery (IGS) systems. These systems have enabled more precise and minimally invasive surgeries compared to conventional surgical techniques [1].

In image-guided neurosurgery (IGNS), accurate and fast optical tracking systems, and registration of preoperative images to the patient allows for the real-time mapping and visualisation of surgical tool positions and orientations with respect to preoperative images, thus guiding the surgeon; similar to a GPS system guiding a driver. One shortcoming of this type of traditional neuronavigational guidance is that the surgeon must shift their attention away from the patient and the surgical field to look at the guidance images on a computer display positioned outside the sterile area. Such shifts can disrupt surgical workflows and be detrimental to the task at hand [2]. Recent work in augmented reality (AR) [3-6] has addressed this issue by providing efficient and intuitive visualisation of the complex 3D patient anatomy within the context of the live view of the operative field (see Fig. 1).

Fig. 1

System set-up: the 3D printed phantom, trackers and the on screen live view of the phantom with vessels extracted from preoperative CTA are shown

The second shortcoming that remains with traditional IGNS systems is the continued loss of patient-to-image registration accuracy which degrades throughout the surgical procedure. Commercial IGNS systems show initial landmark registration accuracies between 2.7 and 6.2 mm, with a median of 4.0 mm, according to a meta-analysis done by Stieglitz et al. [7]. While this is true immediately after the registration procedure, this level of accuracy is no longer observed as soon as the craniotomy is performed. As summarised by Gerard and co-authors [8] in a recent review on the problem of brain shift, most of the studies measuring the problem reported mean shifts in the range of 1–10 mm and maximum shifts in the 10–30 mm range, with up to 50 mm [9] of shift. This makes brain shift the largest contributor to registration error; much higher than that of all other sources of errors combined, including errors arising from technical inaccuracies in the tracking, distortion in the preoperative images or initial registration error. Its impact is so large that many surgeons use IGNS systems to approach a surgical target but stop using it during the procedure when the registration accuracy has degraded too much.

Albeit being the biggest limitation affecting IGNS systems, there is still no truly satisfying solution to solve the problem of misregistration. Brain-shift is a complex phenomenon with multiple causes, making it hard to compensate for in an automated fashion. Many attempts have been proposed using either intraoperative imaging [10-13], where intraoperative images are re-registered to preoperative ones or biomechanical models [14, 15], where the aim is to predict the expected displacement using a patient-specific physical model. While these methods show promise, it is also clear that the road ahead to make these methods more robust, more general and less sensitive to the occasional sparseness of intraoperatively acquired images and data will be long and strewn with pitfalls.

While there is much research on using intraoperative imaging or modelling to account for brain shift, simpler methods that aim to give the surgeon control over the registration not only at the beginning of surgery but also during surgery, have not fully been explored. In this Letter, we present a method to rigidly re-register images at any point in surgery using touchscreen gestures (i.e. panning, rotation) on a tablet showing an augmented reality view of the surgical scene. The system allows the user to both translate and rotate the virtual preoperative patient images (visualised using AR) to the actual real-time images of the surgical scene. Rotation of the images is done with two fingers around the optical axis of the tablet's camera and the translation is done with one or two fingers parallel to the camera image plane. Since the mobile device can very easily be moved around the patient, the user can translate in any plane and rotate around any axis. It thus gives access to the full range of rigid transformations to the surgeon. This method does not aim at replacing more complex non-rigid registration correction methods, such as FEM modelling or intraoperative imaging, but rather at complementing them. Our method is much simpler to use and has a negligible footprint intraoperatively, both time-wise and resource-wise. Thus, it could be used to make a quick rigid registration correction when time or resources are limited.

Related work

The method we propose has a similar goal to the ones recently presented by Drouin et al. [16] and Kantelhardt et al. [17], which to our knowledge are the only manual registration correction methods in the literature. Drouin et al. proposed a method to perform manual re-registration by using a tracked pointer to trace vessels using both the virtual patient data in an AR view (where the live video was captured by a neurosurgical microscope and the AR visualisation was displayed on the computer monitor of the IGNS system) and on the actual patient cortex. Given the two sets of user defined vessel line traces, ICP (iterative closest point) was used to find a rigid transform between the points on the lines in order to correct for registration errors. Their results showed that users were able to correct for most of the registration error for medium to high shift cases, but degraded the registration for smaller shifts. They obtained a mean RMS error after correction of 4.06 ± 0.91 mm.

In Kantelhard et al.'s work, the user can translate the pre-operative patient images in the x and y directions using arrows on the computer screen. Kantelhardt's method allows only for translation in the plane of the microscope, which is a strong limitation of the method. Considering this last point, we will use mostly Drouin et al.'s method as a basis for comparison to our method.

Our proposed method, contrary to Drouin et al.'s, does not make use of the tracked surgical pointer and offers in-situ AR visualisation. While it is true that for microscope-guided navigation the surgeon might already have the tracked pointer handy, it would not necessarily be the case for mobile AR-guided navigation. Indeed, with an in-situ AR display, pointer free navigation is possible and has been shown to positively affect the surgical workflow by avoiding disruptions and limiting the number of times the surgeon must use the pointer to correlate guidance images to the patient [18]. We believe, therefore, that in this context our method would integrate more seamlessly into the surgical workflow and be more intuitive and simpler to use.

In this study, we tried to assess if our method would enable a similar accuracy to Drouin et al.'s while using a simpler interaction method. In addition, advantages over previous methods highlighted by Drouin et al. still hold for our method, namely: results are robust since they make use of the surgeon's extensive knowledge of anatomy; the method is quick and easy to use, making it possible to re-register as often as needed during a procedure; and the transform found with our method could be used as a starting point for more complex and automated methods, thus reducing the risk of getting stuck in local minima. Further advantages of our method over previously proposed ones are (i) the ability to easily compensate for shifts in all directions, not only in-plane ones (since the device can very easily be moved around and has much more freedom than the microscope); and (ii) the use of simpler gesture-based interactions directly on the mobile device where the AR is displayed, which allows the surgeon to do the registration without relying on a technician to interact with the navigation system, nor using the surgical pointer.

System description

Our mobile IGS system comprises of a Polaris Tracking System (Northern Digital Technologies, Waterloo, Canada), an iPad (Apple Inc., Cupertino, USA) with MARIN: Mobile Augmented Reality Interactive Neuronavigator, a developed AR App, the IBIS (Intraoperative Brain Imaging System) Neuronav open-source platform for image-guided neurosurgery [19], along with a wireless router to relay data to and from the iPad to IBIS. The router through which both devices send video frames and commands is a TP-Link TL-WR810N, which uses the IEEE 802.11n wifi standard. The mobile application was built upon improving the system presented in [2]. IBIS runs on a desktop computer with an i7-3820 3.6 GHz CPU, NVIDIA GTX670 GPU, ASUS PCE-AC55BT (Intel 7260 chipset) wireless PCI card and Ubuntu 16.04.4 LTS (with the latest available wireless drivers (iwlwifi 4.13.0-43). The iPad used is an A1893 (6th generation) model, with the Apple A10 Fusion system-on-chip 64-bit architecture, an 8.0 MP camera and iOS 11.3. The iPad was outfitted with a passive tracker that was attached to a custom 3D printed case (see Fig. 1).

The IBIS Neuronav package comes with plug-ins for tracking, patient-to-image registration, camera calibration, and the capability to use augmented reality visualisation by capturing a live video stream from a microscope or video camera and merging it with preoperative images on the monitor of the system itself. In our work, we extended the IBIS Neuronav system to allow for augmenting an image, not only on the monitor of the system but on a mobile device (i.e. a smartphone or tablet) that captures the surgical field of view, thus allowing for in situ AR visualisation, and the integration of gestures. The system set-up as used in the lab on a 3D printed phantom head is shown in Fig. 2.

Fig. 2

System setup: tracking camera in the top-left corner, IBIS workstation in the lower-left corner and tablet in the lower-right corner with a phantom

To make use of IBIS’ existing functionality, the mobile device (i.e. iPad) serves merely as a camera and display. The costly computations are handled by the desktop on which IBIS runs. In order to create the AR view, we first calibrate the camera of the tablet. Calibration (intrinsic and extrinsic) is done using a modification of Zhang's camera calibration method [20], followed by a second optimisation procedure to find the transform from the tracker to optical centre of the camera (for more details the reader is referred to [19]). The average camera calibration reprojection error obtained was 0.77 mm. Patient-to-image registration is done using a landmark registration, for which we obtained an RMS error of 0.69 mm.

IBIS receives the positions of the iPad and the patient from the tracking camera, then computes the relevant transformations and renders the virtual objects from the camera's viewpoint. The rendered virtual objects are then sent using OpenIGTLink [21] over the local area network to the tablet and blended with the live video feed using OpenGL ES 3.0 and GLSL. The QT framework (version 5.10) was used in the design of the AR application user-interface, and also used in the communication with the mobile camera.

A screenshot of the interface of the MARIN App is shown in Fig. 3. Options for the registration correction module are displayed in a panel at the top-right corner. These options serve as switches to turn the rotation and translation mode on or off and to perform either both at the same time or only one at the time if needed. The last button in this panel is a reset button, which allows the user to restart the correction procedure from the initial misregistered image.

Fig. 3

Screenshot of the iPad AR application during use. Options for the registration correction module are displayed in the top-right corner. The green and yellow switches allow the user to turn rotation and translation modes on or off to perform either both at the same time or only one at the time. The last button on the right is a reset button that allows the user to restart the correction procedure from the initial misregistered image

The touchscreen gestures registered by our app are (i) panning and (ii) pinching with rotation, both of these are common gestures used in mobile interfaces and should thus be intuitive to users. Panning results in a translation of the objects in the plane of the camera image. Pinching with rotation, also sometimes simply referred to as rotation, results in a rotation of the object around the optical axis of the camera. Both gestures result in a transform that is directly applied to the virtual object. Therefore, users have access to the full range of rigid transforms by compounding translations and rotations from different perspectives.

Methodology

Testing of the usability and accuracy of our system was done in a user study where participants were presented with a misregistered AR view and were asked to correct the registration with our system. More information on the study design and sample is given in Section 5.

The phantom on which the study was performed is a 3D nylon printed head whose model was extracted from MRI and CT angiography (CTA) data of a patient operated on for an arteriovenous malformation (AVM). Two simulated craniotomies expose the cortex and surface vessels: one in the posterior parietal lobe, centred on the AVM, where multiple vessels are visible and the other one in the temporal lobe, where only a few vessels were visible. At the corners of each of the square-shaped craniotomies are four landmarks, which were used for the initial registration, and the determination of the RMS error. See Fig. 1, e.g. view of the system during the experiment.

A close-up of vessels near the AVM in the first craniotomy is shown in Fig. 4. In the bottom two images, an example of registration correction performed with the system is given. On the left, we can see vessels translated from their correct position, while on the right they have been realigned. Note that edges in the camera image are detected and used to modulate the blending coefficient between the real and virtual images in the AR view. The augmentation opacity on edge pixels is reduced to allow the user to see boundaries of real objects. These boundaries, as we can see in the bottom-right image of Fig. 4, should line up with the boundaries of the virtual objects when images are properly registered. This cue could thus be used by users to guide them during the manual registration correction procedure.

Fig. 4

Augmented reality visualisation. Top-left: Phantom without augmentation. Top-right: Virtual image of vessels. Bottom-left: Augmented reality view where the volume is misregistered, as seen on the tablet. Bottom-right: Augmented reality view where the volume has been re-registered, as seen on the tablet

Note that in the user study, we performed, the displayed mesh was that of only surface vessels of the relevant hemisphere. It was thus similar to what would typically be shown to a surgeon for an AVM procedure on an AR overlay.

The measured outcomes of the study were registration RMS error after re-registration and percentage correction from the initial misregistration. Further, the total time to re-register the images and the number of times subjects moved the tablet around the phantom to look at the misregistraton from different perspectives was measured.

Experiment

A protocol as similar as possible to Drouin et al.'s was chosen, in order to be able to compare our method with their pointer based re-registration method. As in Drouin et al.'s validation, subjects were presented with five misregistrations to correct, all of the same preoperative images and on the same phantom. Initial registration offsets were the same as in Drouin et al.'s evaluation and are given here in Table 1. Offsets were generated by translating in a plane with a given amplitude, in a random direction and rotating around the optical axis with a given angle, in a random direction. Both translation amplitude and rotation angle decreased with every trial.

Table 1

Initial registration offsets

Trial no.	Translation amplitude, mm	Rotation amplitude, deg.
1	15	5
2	10	4
3	6	3
4	3	2
5	1	1

Our study sample was composed of ten subjects (aged 24–57 (median 29), 7 males and 3 females). They were students, professors, engineers and neurologists, all working in either biomedical engineering, computer science, or medical image analysis. All subjects were already familiar with neuronavigation systems and augmented reality. Nevertheless, they were re-familiarised with the concepts and the method's intended usage prior to the study. Subjects were then briefed on the system functionality, interaction modes and study procedure. After the instructions, but before the beginning of the trials, subjects were presented with a correctly registered AR view and given time to become familiarised with the system. They were asked to use the touchsreen gestures to move the virtual images around and to re-register them as accurately as possible. This pre-test trial served as a practice and to help reduce potential learning bias. It was emphasised to the subjects that the goal was to be as precise as possible and make the best registration correction as possible, for which they could take as much time as they felt was necessary.

After the evaluation, subjects were asked to fill out a questionnaire consisting of demographic questions, the System Usability Scale (SUS) questionnaire [22] and additional questions on the method's usability. The SUS is a standardised usability questionnaire consisting of ten five points scale questions. It has been used on a large number of applications and interfaces, for which data is available. It is, therefore, a well-established test to estimate a system's usability and compare against other systems. A system scoring above 68 is considered usable and above average.

Results

Before reporting any measure, it is worth mentioning that two individual trials were flagged as outliers. Their registration RMS errors after correction were respectively 7.7 and 10.5 standard deviations away from the distribution mean. Considering that these two trials (coming from two different subjects) were much further away from normal performance than any other trial both across all subjects and within each of the subjects, we believe that there was an experimental error or technical issue on those trials. We, therefore, removed these two trials from all further analysis.

The median registration RMS error after correction for all subjects on all trials was 4.13 mm. Interestingly, if looking separately at the landmarks around the two distinct craniotomies, we observe that the median registration RMS error distances are significantly different: 3.51 mm (for the craniotomy of interest where the AVM and many other vessels were visible), and 4.88 mm for the second cranitotomy. A one-way ANOVA comparing the registration RMS error of the landmarks around each of the craniotomies shows that there is a statistical difference between the RMS error means . Landmarks around the prevalent craniotomy will be used in further analysis.

Although we believe that the median is a better outcome measure in this case since the distribution is not Gaussian, we still computed the mean, in order to compare against Drouin et al.'s method. We obtained a mean registration RMS error after correction of 3.84 mm with a standard deviation of 1.34 mm, comparable to Drouin et al.'s error of 4.06 mm with SD of 0.91 mm. The RMS error after correction against the initial misregistration RMS distance, for comparison with Drouin et al.'s method, is shown in Fig. 5. Distribution of the points and linear regression fit are comparable to what is reported in [16].

Fig. 5

Data points for all trials and all subjects presenting the relationship between the initial registration RMS error and the registration RMS error after correction (blue dots) with its linear regression fit (red line) and the zero correction partition (green dashed line)

A more telling measure of performance is the percentage of misregistration RMS error that subjects were able to correct for as shown in Fig. 6. This graph makes it easier to distinguish between negative and positive performances. This difference can also be seen in the previous graph, where points either lie under or below the zero correction partition (green dashed) line. As we can see from the graph, when the initial misregistration was under the median RMS error of 3.51 mm subjects had a negative performance, degrading the registration error. However, in looking at only medium to high shifts (i.e. an initial misregistration RMS >4 mm), which would be the target usage for the method, subjects achieved a median percentage correction of 55%.

Fig. 6

Data points for all trials and all subjects presenting the relationship between the initial registration RMS error and the percentage of RMS error corrected (blue dots). The median RMS error after correction is indicated by the green line. Data points under the grey line (equivalent to 0% correction) indicate negative performance, i.e. the subject made the registration worse. Data points above the zero correction partition line indicate the percentage by which the subject improved the correction

The time taken to complete the correction ranged from 11 to 161 s, with a median 52 s. The number of tablet displacements varied a lot from subject to subject (range 0–10) and seemed to be more of a personal preference. There were no significant interaction effects found between the time taken, number of tablet displacements, and accuracy. The Pearson's correlation coefficients for time and number of displacements with final registration RMS error are 0.13 and 0.05, respectively, indicating that neither have an influence on the accuracy. Looking at those two factors against initial registration error, the Pearson's correlation coefficients are 0.02 and −0.15, respectively, showing here that the initial offset also does not influence the required time to complete the correction or the number of required movements.

In terms of qualitative evaluation, a few comments came out of the usability survey we gave our study participants. Three shortcomings in the interaction were perceived by users as having a negative impact on the attainable accuracy: jitter, latency and sensitivity. Jitter is an artefact of the tracking system that causes the AR overlay to shift slightly due to inaccuracies in the infrared reading. It is entirely dependent on the tracking system and is thus not particular to our method. Latency is the time between the moment the subject makes a gesture and the moment the images get updated. This latency was caused by the compounded effect of many factors, including network transfer of the command and of the augmentation images and was in the 100 ms range. Sensitivity is considered the ratio between gesture amplitude and resulting transformation amplitude. Despite these shortcomings, the mean SUS score across subjects for our system was 70.5, which is above the average of 68 indicating a usable interface and system.

Discussion

The first thing that stands out when looking at Fig. 6 is the clear distinction of subjects’ performance between trials where the initial RMS registration error is above and below 4 mm. When above, subjects were almost systematically able to correct for the most part of the shift. Although, when below, subjects usually deteriorated the registration accuracy. This would seem to indicate a bound on the attainable accuracy with our system in its current state. Looking back at Drouin et al.'s results, a similar trend can be noticed. We posit therefore that this is perhaps a limitation of the study design. Further, it is not clear that this deterioration would translate into clinical practice. In the setting of this and Drouin et al.'s study, subjects had to try to correct the registration, even when it was only very slightly offset. In clinical practice, we believe that surgeons would only use the correction method when they feel the preoperative images have suffered a visually significant shift, typically several millimetres, and the registration is no longer accurate enough for guidance. Thus, in those cases where the initial RMS offset was below 2–4 mm (i.e. a typical commercial system registration error), for most surgeons probably, it would have been deemed usable and re-registration would not be done.

The second distinction to raise between our study and clinical practice is the user's prior knowledge. We only enrolled participants who were familiar with AR, IGNS, and medical imaging data, which made them similar to surgeons in that respect. However, many of them were not necessarily experts of neuroanatomy, contrary to surgeons. Furthermore, they had not done preoperative planning on the images and thus were not familiar with them, unlike in real clinical cases where the surgeon would already be familiar with the anatomy. Considering those two points, we believe that the median accuracy obtained from our subjects in this study could be considered to be a lower bound on what surgeons would be able to achieve.

Our method produced a mean registration RMS error of 3.84 mm compared to Drouin et al.'s result of 4.06 mm. While our result is slightly better, it is hard to say if our method really offers better accuracy considering the relatively small sample size of both studies (ten in our case and five in the case of Drouin et al.). A direct comparison of both methods with a larger sample size would have to be done in order to gain a better understanding of the potential differences in attainable accuracy. With the current data, we can at least say that they have comparable levels of precision. In addition, the linear regression fit for both samples seems to also be comparable.

We posit that the larger error on landmarks further away from the craniotomy of interest resulted from the relatively large difference that a slight error on the rotation causes on points far from the centre of rotation. This error, however, may not be clinically as important as ones close to the site of interest. This is in line with what real use cases would be. Surgeons would typically be interested in targets close to the site of the craniotomy and would not necessarily be much affected if a slight rotation caused landmarks on the opposite side of the cranium to be slightly misplaced. Further, it is deemed that in real cases, surgeons would only use the system if they felt that enough features were visible to allow for the correction to be done. In addition, surgeons who have better knowledge of anatomy may be able to pick up smaller features to match than non-experts.

Another interesting point to come out of the data is the very low time taken to perform the correction. All corrections, even though subjects were instructed to take as much time as needed to perform the best registration possible, were done under 3 min and the median time to correction was under a minute. The short time requirement as well as the fact that our method can be performed by surgeons themselves without the intervention of a technician hints that the disruption caused by our method to the procedure is minimal. In addition, since the tablet can be draped in a sterile bag, our method can easily be used throughout surgery and as often as needed. Although, and as alluded to in the introduction, this method would not substitute itself for more complex non-rigid brain-shift compensation methods, but it could be used early on and in between uses of those methods. Indeed, non-rigid compensation methods are computationally expensive and require usage of a form of intraoperative imaging, whether it be ultrasound, intraoperative MRI or CT, which makes them much more heavy to use both time-wise and resource-wise. Thus, the real potential of our method lies not in contrast with more complex methods, but in complementarity with them.

As discussed above, time was not correlated with registration accuracy or amplitude of the initial shift. This would hint at two interesting findings. First, it provides an incentive to use this method to compensate for large shifts since the time requirement stays the same and is very small regardless of the amplitude of the shift. It further motivates the use of the method as a first correction before using a more complex and automated method to compensate for most of the rigid misregistration in less than a minute in order to ensure that an automated method would not get stuck in local minima. Second, regarding the perhaps surprising absence of correlation between time spent and final accuracy achieved, we posit that there is an upper bound in terms of registration accuracy achievable with the method. However, this bound can be achieved within a short amount of time.

Regarding subjects’ comments, the two factors specific to our method that was seen as negatively impacting accuracy, namely latency and sensitivity, could be improved in a future version of the system. Network bandwidth is currently the largest contributor to latency. Latency could thus be reduced by optimising compression in image transfer and upgrading the router and network card of the workstation to the newer 802.11ac wifi standard. Sensitivity would also be increased, but perhaps even better would be giving the users control by allowing them to set the sensitivity themselves from within the application's interface. This would allow users to quickly move the volume in its approximate position and then increase sensitivity towards the end to gain finer control and potentially reach a better registration accuracy.

Another interesting comment from users was about the axis around which the volume is rotated. In the present version of our method, the volume is rotated around the optical axis of the camera. Some users, however, mentioned that it may be easier if the rotation was done around an axis parallel to the optical axis of the camera, but translated in the plane of the camera to pass through the centre of the volume. We did not picture this as a potential problem since we thought users would usually have the object of interest more or less in the centre of the screen, in which case those two axes would coincide. However, this was not always the case.

A final comment raised by participants, which seemed of particular interest for future revisions of our proposed method is the ability to translate in the z-direction without having to move the tablet. In the current version, if a user wants to translate in z, they must first physically move 90° around the patient's head, and translate in the plane by the desired distance (in what is now the x or y direction), then return to the initial position to verify that the transformation is correct. This can be time consuming and often requires moving the tablet many times in order to obtain the desired correction. It was suggested by users to add a pinch gesture to translate along the z-direction of the current view, thus enabling quicker correction with less movement.

The usability score of 70.5 we obtained for the SUS places our system slightly above the average for the test of 68. While not very high, it certainly shows, that even in the present state, the system is usable. Although, it can be assumed that after integrating the user's comments and suggestions, a further version would be much more usable and also potentially perform better in terms of accuracy.

Conclusion

This study shows that using gestures on a mobile device to correct the registration error of an AR IGNS is a valuable option. Users were able to achieve accuracy comparable and even slightly better than previously proposed methods. The collected data further suggests that a sufficient number of image features (vessels, in our study) would be necessary in order for subjects to achieve a valid correction. For other types of procedures where angiography data is not available, cortical features could be matched. Results also show that, similar to previous methods, the relevance of the method varies with the amplitude of the initial misregistration. Indeed, for medium to large initial shifts (i.e. >4 mm), subjects were able to account for most of the error, but for smaller shifts, subjects were usually unable to improve the registration.

The present method's usability could be improved by integrating comments from users such as adding interaction modes and tweaking interaction parameters. It could be hypothesised that enhancement in usability would lead to enhancement of achievable precision. In addition, more image data sources could be added to the AR view and combined in a coherent model, such as the cortex surface in addition to vessels. Considering how well acquainted with the anatomy surgeons are, the more data that is made available to them the more likely they would be to find helpful features for the correction, conditional to maintaining a clear and uncluttered view of the data. Although the iPad AR system itself has been brought into the operating room for initial testing where we have received positive feedback, we have yet to test the registration feature. Once the improvements mentioned above are implemented in a future revision of the method, they will be tested with surgeons during real clinical cases in the operating room.

Funding and declaration of interests

This work was supported by NSERC (the Natural Sciences and Engineering Research Council of Canada) and the Faculty of Engineering at Concordia University.

Conflict of interest

None declared.