Augmented reality in scientific visualization and communications: a new dawn of looking at antibody interactions.

The use of augmented reality (AR) in providing three-dimensional (3D) visual support and image depth have been applied in education, tourism, historical studies, and medical training. In research and development, there has been a slow but growing use of AR tools in chemical and drug discovery, but little has been implemented for whole 3D antibody structures (IgE, IgM, IgA, IgG, and IgD) and in communicating their interactions with the antigens or receptors in publications. Given that antibody interactions can vary significantly between different monoclonal antibodies, a convenient and easy to use 3D visualization can convey structural mechanisms clearer to readers, especially in how residues may interact with one another. While this was previously constrained to the use of stereo images on printed material or molecular visualization software on the computer, the revolution of smartphone and phablets now allows visualization of whole molecular structures on-the-go, allowing rotations, zooming in and out, and even animations without complex devices or the training of visual prowess. While not yet as versatile as molecular visualization software on the computer, such technology is an improvement from stereo-images and bridges the gap with molecular visualization tools. In this report, we discuss the use of AR and how they can be employed in the holistic view of antibodies and the future of the technology for better scientific communication.

Statement of Significance

Recent technological progress has allowed augmented visualization of three-dimensional antibody structures using mobile devices. This allows an on-the-go convenient visual appreciation of the antibody elements and how the various antibody regions can interact with each other in a new frontier of communicating antibody research that can extend to all structural biology.

INTRODUCTION

The visualization of protein structures is important for scientific communication and drug development methods, such as three-dimensional (3D) pharmacophore modeling [1]. Over the years, molecular visualization software tools like UCSF Chimera [2], PyMOL [3], Rasmol [4], and Cn3D [5] among others have allowed the visual manipulation and viewing of different perspectives of molecular structures. On printed paper, this is limited to stereo-images, requiring advanced stereoscopy eye techniques [6], a skill not everyone can successfully master. While many research articles provide links to downloadable protein complex structure files to be viewed on a computer, this is not possible on printed materials nor easily accessible for scientists without familiarity with the relevant software and their operations.

With recent technological advances in the smartphone revolution, AR is now made available on-the-go in the form of augmented reality (AR) smartphone apps [7], all without the need to pick up technical software operation skills.

As its name indicates, AR incorporates virtual objects in a real physical environment, registering the real objects in 3D in real-time [8]. It was first coined by Tom Caudell and David Mizell in 1992 in a see-through head-mounted display, offering a low cost and efficient alternative in manual manufacturing operations [9]. Following the integration of the global positioning system and the miniaturization of mobile phones as personal digital assistants, the use of AR expanded from navigation systems [10,11] into advertising [12] and games [13]. Utilizing shape-based detection [14] via the smartphone camera motion tracking [15], customization of physical image targets and 3D virtual object visualization can be made and effected directly on smartphones and phablets. By simply pointing the mobile phone camera at the designated physical image (that can be represented by a drawing on a piece of paper or even a card), the 3D model or animation associated with the physical image can be brought on display. Intuitively, rotating the physical image or the smartphone to different angles enables the 3D virtual object to be viewed from different perspectives, including the zooming in or out by moving closer and further from the image target. With the ubiquitous use of smartphones and phablets, AR has been applied to medical training [16,17], science, technology, engineering and mathematics education [18], tourism [19], and heritage tours [20] amongst an inexhaustive list. In R&D, AR fills the gap between stereo-images and molecular visualization tools in structural biology research [21,22]. Given the complexity of biological systems, mobile phone AR apps have edged into the visualization of a large complex involving antibodies, including their interactions with other immune system components in academic publications (see example [23]).

The antibody is a large protein molecule that plays the key role of binding to the antigen and activating the immune system by antibody receptors or other immune proteins. As one of the key adaptive immune response proteins that commonly interacts with multiple partners specifically, its Y-shape is structurally and functionally divided into two: the antigen binding fragment (Fab) forming the “V,” and the rest of the stalk called the Fc which binds to the antibody receptor proteins [24,25] and other immune proteins, such as complement proteins [26,27]. Within the antibody domains, there exists a combination of structural regions (Fig. 1) that can be engineered to avoid undesirable side effects such as immunogenicity, especially when designing therapeutic antibodies. A detailed description to the role of these antibody regions and their functions is discussed in numerous reviews [23,28,29].

Figure 1

A schematic of antibody elements during antibody humanization that can be sagaciously manipulated for incorporating desired features and for avoiding unwanted side effects. From the switching of isotypes to modify the antibody-dependent cellular cytotoxicity (ADCC) and localization, to the choice of Variable Heavy - Variable Light (VH–VL) for production and purification. Augmented reality for this figure (“antibody AR in Sci Comm”) showing all antibodies (IgE, IgM, IgA, IgG, and IgD) can be seen using APD AR Holistic review App (7).

In working toward therapeutic antibodies, sagacious design is important to reduce unwanted side effects that could lead to failure in clinical trials. Such sagacity comes with an in-depth understanding of how antibody regions interact with their binding partners and how the various elements in the other regions of the antibodies can affect the function of other regions. Recent findings have shown that the constant region, although distal, can influence the antigen binding region [30-32] to an extent as drastic as abolishing antigen binding (see an example of IgD in [31]). Such allosteric effects would be better presented in a 3D virtual models for readers than still stereo-images.

The simultaneous viewing of multiple antibodies using stereoscopic images is highly challenging, and not possible with animations. In the absence of good visualizations via virtual platforms [33], pure descriptive passages convey limited structural insights. To overcome this, we describe the use of mobile phone AR technology as a possible easy solution that enables even the visualization of antibody interactions e.g. with receptor (Fig. 3).

Figure 3

Illustration of allosteric communication found between CDRs framework regions (FWRs) and Fc engagement as shown from our previous work [30,32,42]. Augmented reality for this figure (“Antibody Allosteric Comm.”) can be seen using APD AR Holistic review App (7).

MATERIALS AND METHODS

Previously, we reported a brief methodology for making AR [7] for illustrating whole HIV-1 Gag [34] and the use of whole protein structures for analysis [23]. Given the large antibody complexes, the method for AR models viewed using an app needs to be adapted than simply applying what worked for smaller systems. Considering the limitations of a smartphone compared to a laptop/desktop in processing power, mobile apps have additional considerations, such as memory and display screen limitations [35]. It is not easy to display molecular structural details without making the app memory or storage-space intensive, and putting off users of older smartphone models in the downloading of large files or having lagging displays.

We overcame the application size problem in the “APD AR Holistic Review” app by allowing on-demand downloads of the various AR visualizations (Fig. 2) where models that are no longer desired, can be easily removed or re-downloaded again if they are desired. In addition, we also looked into generating smaller file size packages while retaining as much detail as possible (i.e. cartoon and surface representation shown in Supplementary Figure 1).

Figure 2

Snapshots of the user interface in the application. (A) Flowcharts of datafiles from the PDB databank to user download. The .PDB files are processed and exported as .DAE files to retain the color and resolution. Three-dimensional protein models are imported into Unity as FBX files and stored on a cloud server. (B) On-demand download of the AR bundled contents for easy download and removal to accommodate to the users. The AR models are downloaded from a locally hosted database server for recognizing the target image to view the 3D model.

To create the AR, protein structures were obtained directly from online protein databanks (i.e. RCSB PDB) and processed using PyMOL [3], UCSF Chimera 1.11.2 [2], and Blender 2.79 (https://www.blender.org) to generate static 3D protein models as described in our previous work [7]. While the structures were previously exported into a X3D file format to produce a low polygonal mesh of the 3D protein model, allowing a smaller file size download, the recently upgraded “AR Holistic Review” app took a new adapted approach. The structures were first exported to DAE and then processed in Blender 2.79 to reduce the file size of the protein structure and to map the colors designated in the DAE file as a Unity readable FBX file format.

In some cases, specific residues of the structures are differentially highlighted with colors to spotlight residual properties such as conserved scores obtained from ConSurf [36,37] or free energy changes [2]. However, conventional methods of exporting in X3D do not map the colors of the surfaces, thus the DAE file format was chosen to retain the protein surface color. These colors are mapped to the protein surface in Blender 2.79 with the UV unwrapping tool and the in-built Cycles renderer. The 2D image is then wrapped back to the 3D protein model in Unity as a texture surface.

Unity version 2017.3 was used to further animate the 3D models to project a four-dimensional view of the protein with motions. We used the EasyAR package (https://www.easyar.com) and the Image Tracker GameObject in the package to detect the original publication graphic. After bundling the models and the 2D PNG target images together, the bundles are compressed and stored in a locally hosted database server and downloaded onto the user’s mobile phone when initiated.

RESULTS

The well-known adage “A picture is worth a thousand words” has been true to that figures are almost a necessity in scientific publications with some journals requiring a graphical abstract. Yet, a “motion picture” or video, is essentially many frames of pictures changing within seconds, allowing a few minutes of video to tell the story better than one or a few still images. It is in this space that AR can be used to show animations in 3D, from multiple angles and varying magnifications.

The on-the-go visualization of how proteins interact with each other or with small molecules is undoubtedly better presented in a video as opposed to still pictures. Given the restriction of figure numbers in some journals, and that videos cannot be printed on paper, the solution to displaying multiple images or binding sites would be to allow videos to be triggered on ubiquitous personal smartphone devices. Yet, within the implementation of such features, the size of video bundle files (comprising of the 3D model, animations, and target image) has to be balanced considering the quality of the 3D model. In our app, we kept the download size below 20 MB per AR model, while allowing the majority of single-molecule to be displayed in high resolution. We have achieved this even for the interaction of whole antibodies binding to the crystallizable fragment receptor (FcR) (Fig. 3). While there are many proteins of less than 100 residues (i.e. HIV-1 protease) that can be easily displayed in high resolution without taking up a large file size, antibodies (totaling the light chains and heavy chains) have up to 1500 residues per whole molecule. To add to the complexity, displaying multiple antibodies or their interactions with other proteins e.g. antigens or receptors, can further add to the file size and slow down the AR model animation on older devices. By reducing the number of polygons of the antibody model in Blender, we were able to retain the structural information of the protein within a manageable download file size. One such example of multiple whole antibody structures is shown in the AR of Fig. 1.

In an earlier study, we performed multiscale computational simulations on IgM multimeric complexes [38]; however, multiple snapshots of such large simulations are often too large in file size and takes up too much virtual memory on the smartphone, making the simulations slow and unstable. Thus, to provide simulations for antibodies, basic rotations and movements are in place to represent motions, while unanimated stationary 3D structures (e.g. AR of Fig. 1) of such large copies are more feasible for current limitations. It is expected that with the continued increased processing power of smartphones with 5G Internet bandwidth, it is only a matter of time before such simulations of hexameric IgM binding to multiple antigens become common inAR.

While the descriptive texts of protein docking such as “Hydrophobic contacts were observed between L100, K103, V106, Y181, Y188, P225, F227, L234, P236 and Y318 with less prominent interactions between P95, S105 and W229.” [39] are technically correct, it makes very little sense to a reader unless they happen to have a structural memory of the specific protein in high-resolution detail in their mind. While a still figure can still present ideas across in this example, such a figure can only depict a single angle perspective without depth, losing the rotations to see some hidden interactions. There is little doubt that an interactive image or animated AR allows the user to zoom in and out, rotate, change perspective by simply intuitively moving the phone or target image (see https://www.facebook.com/APDLab/videos/3176456715698289/ and https://www.facebook.com/APDLab/videos/2075249849390855/?v=2075249849390855 with reference to the above examples for a full demonstration). For a step-by-step user guide on utilizing the APD AR Holistic review app, see Supplementary Information and https://youtu.be/7kvjkXZ8KYU.

For further illustrations of fine conformational changes such as minor loop movements or single residue mutational effects generated through bioinformatics tools like ENCoM [40] and AlloSigMA [41] without drastic zooming in using AR, protein surface color representation can be used (Supplementary Figure 2) on the returned PDB file from these servers.

DISCUSSION

AR allows easy and “on-the-go” visualization of antibodies leveraging on mobile devices with in-built cameras and fast stable internet access for the initial download of the app and files. Without the constraints of printed space, antibodies are no longer limited to fragments or partial views but instead allows for a holistic view of whole or multiple antibodies (e.g. see IgE, IgM, IgA, IgG, and IgD in the AR of Fig. 1), even when interacting with other proteins e.g. receptors (Fig. 3). Through viewing multiple antibodies simultaneously, a comprehensive visual comparison is made easier. Apart from the gross differences in size and oligomerization, more detailed changes in antibody regions discussed in our prior work [28] can now be easier conveyed. Other potential applications that can benefit from AR visualizations include complementarity determining region (CDR) and even SDR grafting which involve a smaller number of residues than the typical antibody domains.

The upcoming alternative and possibly more immersive visualization method is virtual reality (VR). While VR allows the exciting avenue to walk within and explore molecules as if one was shrunk to atomic size, its immersive nature is also a drawback as it requires the headset and a loss of real-life awareness, making it less convenient than AR to execute on-the-go, thereby restricting its utility in more real-life scenarios.

AR is not without its technical limitations. Inherent file size concerns result from different protein structures. To accurately display protein structures without losing resolution, the AR model must undergo a series of computationally exhaustive checks and compression before display on the smartphone device. Such processes can be challenging for seamless display even with the current latest smartphones. Furthermore, to view interactions at specific antibody–antigen binding sites, significant in-depth magnifications are required. Given the challenges in stable displays of the 3D AR model on hand-held smartphones phones at such magnifications, the fine-tuned pan-and-tilt camera motions at close range are challenging even for the most stable of hands. This stability problem can however be addressed by pinching in and out features for zooming at the price of file size and quality of the 3D model. For our app, we have utilized differently colored surfaces to highlight the residues and interactions to overcome such limitations while displaying interacting regions.

Since protein models obtained from online databanks do not come in differentiating colors, molecular visualization tools like PyMOL [3] and UCSF Chimera that work optimally on higher processing desktops/laptops are still required for initial processing and highlighting of residues before uploading them into apps. As such, AR apps currently do not displace molecular visualization tools but fills the gap between these tools and stereo-images for on-the-go visualization. It is thus likely that there would be more AR in future publications, and it would continue to revolutionize scientific publishing and communication. An AR scientific journal may well be in the horizon.

Click here for additional data file.

Click here for additional data file.