Commentary: What Can Augmented Reality Do for You?

In Hanna et al.'s demonstrative study titled, “Augmented Reality Technology Using Microsoft HoloLens in Anatomic Pathology,”[1] the authors explore different applications of the Microsoft HoloLens, an augmented reality (AR) device, in anatomic pathology (AP). The HoloLens is a lightweight, Internet-enabled headset that “augments” a user's perception of their real-world environment by superimposing the computer-generated images and audio to create a mixed-reality experience. With multiple hand gestures and voice commands, users can annotate, document, photograph, and capture videos of macroscopic specimens; with video conferencing software, the local and remote users can engage in bidirectional interactions with the mixed-reality experience. The authors present six AP use cases including autopsy, telepathology, viewing of three-dimensional (3D) scans of macroscopic specimens, manipulation of whole-slide images (WSIs), specimen radiograph coregistration, and viewing volumetric pathology scans.

In its autopsy application, the HoloLens was successfully used by pathology residents to receive remote dissection instructions from an attending pathologist, and the authors cite an instance in which the pulmonary artery was successfully assessed for emboli using the live video transmission functionality. Similarly, the HoloLens was favorably used for live streaming and bidirectional virtual annotating to support gross specimen telepathology and assist users at the dissection bench. The 3D renditions of multiple macroscopic specimens were viewed and manipulated around multiple axes. With a remote desktop application, users gained access to digital slides stored on the authors' institutional server, and through freeware viewers, WSIs were rendered and manipulated (e.g., scroll, pan, and zoom). At the gross bench, a prosector traditionally correlates radiologic findings by manually detecting calcified regions or metallic biopsy clips and reviewing the specimen radiograph on the workstation monitor retrieved from the picture archiving and communication system (PACS). This conventional method of performing pathology–radiology correlations was compared with an AR workflow using 24 breast specimens, in which the prosector overlays a virtual radiograph atop the macroscopic specimen. For the specimen radiograph coregistration use case, the authors provided objective data showing the AR workflow to be seven times faster than the conventional method, with positive user sentiment. Reconstruction of volumetric data from serial histopathology sections creates large files (upwards of 1 GB), rendering them incompatible with the HoloLens since its 3D viewer applications do not support files larger than 50 MBs.[2] The volumetric data files were optimized and successfully imported in the HoloLens, but could not be manipulated.

COMMENTS

Improvements in AR and virtual reality (VR) technologies have led to their application across various medical specialties in particular surgery and to a lesser extent pathology.[34] Hanna et al. have introduced various applications of an AR device in the AP laboratory, which appears to be advantageous for pathologists over VR setups in that the user is not completely immersed in a virtual experience and rather has a more natural interaction with the real-world environment (e.g., specimen on the gross bench).

Although the HoloLens was shown to be a suitable device for viewing and manipulating WSIs and 3D renders of gross specimens, it is unclear whether users preferred it or would choose it over a conventional workstation monitor. While the HoloLens boasts an improved resolution of 2536 × 1440 compared to previous AR and VR devices, it is only a fraction of what is offered by monitors with resolutions upwards of 7680 × 4320, which may serve as more ideal displays for the interpretation of WSIs.

As demonstrated by Hanna et al., AR shows potential to enhance pathology education since it enables an attending pathologist or senior resident to provide real-time feedback to the prosector on how to approach a dissection or which sections to sample. Since AP is a visual discipline and macroscopic documentation of specimens' features is essential to its effective practice, this type of AR device may also improve the workflow at the gross bench. Photography is a widely accepted method to achieve visual documentation and devices such as HoloLens are not only capable of capturing high-resolution still images but also can record 720p video. The feasibility of sharing digital pathology images for enterprise-wide use into a PACS has been previously demonstrated,[5] and using similar methods may create a thorough electronic health record complete with macroscopic photographs and videos.

The most provocative application of the HoloLens was the specimen radiograph coregistration, which was reported to be highly rated as usable and useful. AP laboratory implementation of an AR solution for real-time pathology–radiology correlation that is similar to what Hanna et al. have described would be a great advance for dissection techniques at the gross bench. Their proposed method has the potential of accurately and quickly identifying radiologic findings on specimens.