Virtual reality and 3D printing improve preoperative visualization of 3D liver reconstructions-results from a preclinical comparison of presentation modalities and user's preference.

BACKGROUND: Preoperative three-dimensional (3D) reconstructions for liver surgery planning have been shown to be effective in reduction of blood loss and operation time. However, the role of the 'presentation modality' is not well investigated. We present the first study to compare 3D PDFs, 3D printed models (PR) and virtual reality (VR) 3D models with regard to anatomical orientation and personal preferences in a high volume liver surgery center.
METHODS: Thirty participants, 10 medical students, 10 residents, 5 fellows and 5 hepatopancreatobiliary (HPB) experts, assigned the tumor-bearing segments of 20 different patient's individual liver reconstructions. Liver models were presented in a random order in all modalities. Time needed to specify the tumor location was recorded. In addition, a score was calculated factoring in correct, wrong and missing segment assignments. Furthermore, standardized test/questionnaires for spatial thinking and seeing, vegetative side effects and usability were completed.
RESULTS: Participants named significantly more correct segments in VR (P=0.040) or PR (P=0.036) compared to PDF. Tumor assignment was significantly shorter with 3D PR models compared to 3D PDF (P<0.001) or VR application (P<0.001). Regardless of the modality, HPB experts were significantly faster (24±8 vs. 35±11 sec; P=0.014) and more often correct (0.87±0.12 vs. 0.83±0.15; P<0.001) than medical students. Test results for spatial thinking and seeing had no influence on time but on correctness of tumor assignment. Regarding usability and user experience the VR application achieved the highest scores without causing significant vegetative symptoms and was also the most preferred method (n=22, 73.3%) because of the multiple functions like scaling and change of transparency. Ninety percent (n=27) stated that this application can positively influence the operation planning.
CONCLUSIONS: 3D PR models and 3D VR models enable a better and partially faster anatomical orientation than reconstructions presented as 3D PDFs. User's preferred the VR application over the PR models and PDF. A prospective trial is needed to evaluate the different presentation modalities regarding intra- and postoperative outcomes. 2021 Annals of Translational Medicine. All rights reserved.

Introduction

In complex liver surgery personalized resection planning is of high importance due to the variability of individual hepatic anatomy (1-3). Hence it is of utmost importance to know the location of the tumor within the liver as well as its positional relationship to the blood vessels. Furthermore, it is necessary to anticipate the sustained vascular and biliary supply and drainage as well as the size of the future liver remnant (1,3,4). To retrieve all this information out of a two dimensional slice-based computer tomography (CT) or magnetic resonance imaging (MRI) is difficult and a mentally demanding task, which needs years of experience and training (5). Three dimensional (3D) reconstructions of the liver parenchyma, the vessels and the tumor have been available for surgical planning for more than two decades (3). Already in 2000, Lamadé et al. were able to show that color-coded 3D reconstructions can increase the precision in tumor localization by 37% compared to the standard 2D CT scan and Lang et al. showed a change of the initial 2D based resection plan due to computer-assisted risk analysis in 33% of their cases (3,6). Within the next 20 years the importance of these 3D reconstructions increased progressively; thus, Banz et al. state that 3D reconstructions are ‘imperative in major, complex liver resections’ and in Japan the universal healthcare insurance covers the additional costs for a preoperative reconstruction since 2012 (7,8). Also several other study groups were able to show that 3D reconstruction is preferable to the standard 2D scans in centrally located hepatocellular carcinoma and bilateral colorectal metastases (9,10).

In the literature, there is no information regarding the ‘optimal’ presentation modality of 3D liver models. Since the beginning of 3D reconstruction and its presentation on 2D monitors as interactive 3D PDFs, technological progress enables not only live-size 3d printed models (PR) but also virtual presentations forms with augmented or virtual reality (AR/VR) devices (11-13). These new technologies (PR and VR/AR) have not been well investigated, especially in comparison with the established presentation mode, even though they have several benefits especially in depth perception compared to a 3D model on a 2D monitor (14). This is the first study focusing on the presentation format of the 3D datasets in hepatic surgery including latest technologies with regard to anatomical orientation and personal preferences in a high volume center for liver surgery. We present the following article in accordance with the MDAR checklist (available at http://dx.doi.org/10.21037/atm-21-512).

Methods

Patient selection and participants

The data of 20 patients with various liver pathologies, who underwent liver surgery in our department, where retrospectively selected for this study (). The selection of patients’ ensured a variety of resection types (minor-/major resection/extended resection/vessel reconstruction). In addition, another inclusion criterion was the availability of a preoperative, high quality CT scan.

Table 1

Characteristics of selected patients (n=20)

Characteristics	Value
Age [years, mean (range)]	61 [22–80]
Gender
Male	15 (75%)
Female	5 (25%)
Underlying disease
Colorectal metastasis	8 (40%)
Hepatocellular carcinoma	4 (20%)
Cholangiocellular carcinoma	2 (10%)
Others (malignant)	4 (20%)
Benign disease tumor	2 (10%)
Operation type
Laparoscopic	4 (20%)
Open	16 (80%)
Vessel reconstruction
Yes	5 (25%)
No	15 (75%)
Type of resection
Minor resection	6 (30%)
Non-anatomical resection (simple)	3 (15%)
Left lateral sectorectomy	2 (10%)
Left lateral sectorectomy and segmentectomy	1 (5%)
Major resection	14 (70%)
Non-anatomical resection (complex)	2 (10%)
Left hemihepatectomy	2 (10%)
Right hemihepatectomy	3 (15%)
Extended left hemihepatectomy	3 (15%)
Trisectorectomy	1 (5%)
Mesohepatectomy	2 (10%)
Associating liver partition and portal Vein ligation for staged hepatectomy (ALPPS)	1 (5%)

A total of 30 participants of different training levels [medical students (n=10), residents (n=10), fellows (n=5) and HPB experts (n=5)] were included in this study.

Three-dimensional models

For three-dimensional reconstruction, a semi-automatic, server-based program (Synapse 3D, FUJIFILM Europe GmbH, Düsseldorf) was used. The three-dimensional visualizations were based on the preoperative computed tomography (CT) scans. A surgical resident of the department (FH), who has five years of clinical experience in a surgical high volume liver center and three years of experience using the reconstruction software, performed all reconstructions (15). These reconstructions can be directly exported from the software to a 3D PDF as well as to stereolithography (STL) files, which are required for the 3D PR models as well as the VR application. Therefore, all examination modalities are based on the same reconstruction ().

Figure 1

3D reconstruction of a patient’s individual liver with three colorectal metastases located in segment 5 and 8 as a 3D PR model (A), as 3D PDF (B) and in VR (C).

PDF: the 3D PDF is directly created by the software via an export function. Within the PDF the liver model can be rotated around a vertical or horizontal axis via the keyboard or manual in all directions and differently scaled using the mouse. In addition, the different structures (e.g., tumor) can be displayed solid, transparently or hidden.

PR: The 3D models were printed by Cella Medical Solutions (Murcia, Spain) using the STL files, which are generated by the reconstruction software as previously described (16). For the parenchyma a transparent material (polyurethane rubber) is used, so that the colored structures (acrylonitrile butadiene styrene with pigments) in the inside (e.g., vessels, tumor) can be seen. The solid and inflexible liver can be held and turned by hand enabling haptic feedback.

VR: The specifically developed application, using the game engine Unity (version 2019.2.14f1, Unity Technologies, San Francisco, California, USA) enables an easy upload of STL files via drag and drop and then automatically generates the VR 3D model. The application provides native support for all OpenVR compatible VR headsets. The 3D liver model can be moved, rotated around a vertical or horizontal axis as well as manual by means of the VR controller. Additionally, the model size can be scaled and the structures can be continuously adjusted to different degrees of transparency (solid to hidden) (17).

Equipment, tests and questionnaires

gives an overview of all used tools.

Table 2

Compilation of used equipment, test and questionnaires

Tool	Specific description
Reconstruction software	Synapse 3D (FUJIFILM Europe GmbH, Düsseldorf)
VR glasses	HTC Vive (HTC Corporation, Taoyuan City 330, Taiwan)
Laptop	MSI Gaming GT63 (Micro-Star int’l Co., Ltd., New Taipei City 235, Taiwan)
Desktop	Fujitsu Esprimo P958 (Fujitsu Technology Solutions GmbH, Tokyo, Japan)
Monitor	24” LG-24BK550Y-B (LG Electronics, Seoul, South Korea)
Stereoacuity test	Titmus test (Stereo fly test, stereo optical company, Inc., Chicago, USA)
Mental rotation test	Hose figure test (18)
Usability test 1	System Usability Scale (19)
Usability test 2	Short User Experience Questionnaire with assessment of pragmatic quality and hedonic quality (20)
Vegetative symptoms test	Simulator Sickness Questionnaire (21)

VR, virtual reality.

Study protocol

The study consisted of two sessions (). At the beginning of the first session a standardized study description and explanation was given to all participants, followed by two tests regarding their spatial thinking and seeing [Hose figure test and Titmus test (18)] as well as the pre-simulator sickness questionnaire (21). After a short introduction of the presentation modalities the participants had to name all tumor-bearing segments for each liver model. The time needed for the assignment was recorded. The participants received no feedback regarding the correctness of their answers. To minimize a selection or recognition bias the modality was changed after every fifth liver model (= block) and each participant had an individually randomized order of presentation modality and liver ID. The first session was terminated after the first 30 models were processed. The post-simulator sickness questionnaire was completed after the first VR block.

Figure 2

Study design. SSQ, Simulator sickness questionnaire; SUS, System Usability Scale; UEQ, User Experience Questionnaire.

The second session started with the assignment of the remaining 30 models in the same way as in session 1. At the end the participants were asked to answer several questionnaires regarding usability and personal preferences (19,20).

Tumor assignment

To establish a standardized correct answer, an internationally well-known expert in the field of liver surgery (HL) with many years of experience regarding hepatic tumor assignment evaluated the CT scans of all patients prior to the study. The assigned tumor-bearing segments based on the 2D data were then reevaluated by the expert using the different 3D modalities. The assignment was based on the Couinaud classification (22).

Statistical analysis

All statistical analyses were conducted using IBM SPSS Statistics 26 (IBM, Armonk, NY, USA) and R v. 3.6.2 [R Core Team (2019), R: A language and environment for statistical computing, R Foundation for Statistical Computing, Vienna, Austria]. The Mann-Whitney U test was used for continuous, unrelated data and the Friedman test for continuous related data. Spearman’s rank correlation coefficient was used for correlation analysis. The impact of different variables on time and validity was analyzed using mixed models with liver ID and participant ID as random variables utilizing the lme4 R package (23). Time was analyzed on log scale using the lmer function. Validity was measured as proportion of correctly detected liver segments and analyzed using the glmer function with binomial family and the number of segments as weight. A correctness ratio was calculated by dividing the number of correct assigned segments through the number of segments. The local significance level was set to 5%. Correction for multiple testing was not performed due to the exploratory character of the study.

Ethics

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the regional Ethics Committee of the Medical Association of Rhineland-Palatinate, Germany (NO.: 2019-14449) and informed consent was taken from all individual participants. The study participation was voluntary and an expense allowance was offered to all participants.

Results

Demographics

Gender distribution over all qualification levels was even (m/f =15/15): medical students (m/f =3/7), residents (m/f =5/5), fellows (m/f =4/1) and HPB experts (n=5; m/f =3/2). Righthandedness was predominant (n=27).

Random effects for the correctness ratio reveal a low variance between participants (var =0.041) and high variance between livers (var =0.996). This implies that tumor assignment of some liver models was more difficult than others for all participants regardless of the level of education.

Participants named significantly more correct segments, when the liver model was presented in VR (P=0.040) or PR (P=0.036) compared to PDF. No significant difference was seen between VR and PR (P=0.964) (). Also the level of qualification had an influence on the correctness of tumor assignment. HPB experts were significantly better than students (P<0.001) by achieving a mean ratio of 0.87 (SD 0.12) compared to students 0.83 (SD 0.15) analyzed over all modalities. gives a detailed summary of the different correctness ratios categorized by modality and level of education.

Figure 3

Mean number of correctly named segments. The diamond indicates the mean number of correct segments for all participants and all liver models grouped by presentation modality with a statistically significant difference between VR and PR compared to PDF. The whiskers represent two standard deviations above and below the mean.

Table 3

Comparison of mean achieved correctness ratios for assessment of tumor-bearing segments for different education levels

Presentation modality	Correctness ratio				P1
	Students, mean [SD]	Residents, mean [SD]	Fellows, mean [SD]	HPB experts, mean [SD]
3D VR model	0.84 [0.14]	0.86 [0.13]	0.84 [0.16]	0.88 [0.12]	0.121
3D PR model	0.84 [0.14]	0.86 [0.13]	0.85 [0.14]	0.87 [0.11]	0.149
3D PDF	0.82 [0.16]	0.85 [0.15]	0.83 [0.16]	0.87 [0.13]	0.038*

1, Kruskal-Wallis test; *, Significant; HPB, hepatopancreatobiliary; SD, standard deviation; 3D, three-dimensional; VR, virtual reality; PDF, portable document format; PR, printed.

In contrast to the time analysis, the result of the hose figure test had a significant positive influence on the correctness ratio (P=0.011; ), indicating that participants with higher abilities for 3D thinking are better in the assessment of tumor-bearing segments regardless of presentation modality. Also better Titmus test results had a positive influence on the correctness ratio, but fail to reach statistically significance (P=0.290; ) for univariate analysis. However, in a multivariate analysis, test results for stereoacuity test show next to presentation modality, qualification of participants and the hose figure test a significant influence (P=0.019). Gender had no influence on assessment quality (P=0.393). Compared to time analysis, where participants were significantly faster in the second study session, this had no influence on the assessment of tumor-bearing segments (P=0.487), implying that short “training” during the first session improves average speed but not quality.

Figure 4

Influence of spatial thinking and seeing on the correctness ratio. (A) The diagram displays the influence of the test results of the mental rotation test on the correctness ratio for each modality. (B) The diagram displays the influence of the stereoacuity test results on the correctness ratio for each modality.

In addition, analysis of time to correctness ratio reveals a low negative correlation (P<0.001, rho =−0.229). This indicates that it is in general not helpful to prolong the assessment time to achieve a higher assessment quality, meaning that the participants either identify the tumor-bearing segments quickly and correct or have difficulties to name the affected segments regardless of assessment time.

Time analysis

Random effects for time show a moderate variance between participants (var =0.12) and between livers (var =0.27).

Comparing the time needed to assign all tumor-bearing segments 3D PR was significantly faster than PDF (P<0.001) or VR (P<0.001). No difference was seen between PDF and VR (P=0.793). In conclusion, the total time needed to assign the tumor location for liver models was significantly shorter for the 3D PR (509±156 s) compared to the VR (702±248 s) and the PDF (734±281 s) (P<0.01).

Regardless of the presentation modality, HPB experts were significantly faster than students (P=0.014; ). No statistical significant difference was achieved between the other subgroups (P>0.05).

Figure 5

Boxplot. The diagram displays time needed for tumor assignment for each modality grouped by qualification level. The assessment time is displayed as log scale. (*P<0.05).

Univariate analysis shows no relevant influence on time neither for the test result of spatial thinking nor of visual spatial ability or gender (P>0.05). Multivariate analysis confirmed a statistical relevant influence of study session and modality on time (P<0.001).

Questionnaires and tests

The new VR application was rated best regarding usability for preoperative liver surgery planning. Ninety percent (n=27) stated that this application can positively influence the operation planning. The 3D PR was rated as helpful by 73.3% (n=22) and the 3D PDF by 46.7% (n=14). In addition, 73.3% (n=22) preferred VR over the other modalities and 26.7% (n=8) favored the 3D PR. The selection of preferred modality was influenced by the level of education. So favored 100% (n=10) of students and 80% (n=8) of residents in comparison to 40% (n=2) of fellows and HPB experts the VR application. The PR on the other hand was the preferred modality in the group of fellows and HPB experts with 60% (n=3) compared to 0% of students and 20% (n=2) of residents, respectively. This results in a statistically significant difference between experienced (fellows and HPB experts) and rather inexperienced (students and residents) (P=0.028).

Twenty-five participants (83.4%) rated the VR application and 22 (73.4%) the 3D PR model as good or even excellent in the System Usability Scale, compared to just 10 (33.3%) for the 3D PDF. The mean score values [SD] were: VR 83.25 [13.99], 3D PR 80.25 [13.22] and 3D PDF 61.92 [21.57].

The mean score and subscore of the User Experience Questionnaire are displayed in . Compared to the benchmark values of Schrepp et al. the VR application achieves excellent results (in the range of the 10% best results) and the 3D PR models are in all categories above average (25% of results better, 50% of results worse) whereas the 3D PDF is rated below average (in the range of the 25% worst results) (24).

Figure 6

Boxplots of subscores and score values of the User Experience Questionnaire (UEQ) for the three different presentation modalities. Values above 0.8 are defined as positive evaluation results (red dotted line).

Comfort of the VR glasses was rated good or very good by 93.4% (n=28) of participants. Results of the Simulator Sickness Questionnaire show that the use of the VR application induced no relevant vegetative sight effects [mean (SD): total scoreprae 58.49 (67.92) vs. total scorepost 89.50 (148.19); P=0.475].

Discussion

In complex liver surgery a profound knowledge of the patient’s individual hepatic anatomy is of particular importance. 3D reconstructions of the liver have been shown to be helpful for resection planning. Several studies have demonstrated that 3D visualizations have advantages over 2D scans (5,25,26). However, the role of the ‘presentation modality’ for these 3D visualizations is not well investigated. Our results indicate that the precision of tumor localization is higher when the reconstruction is presented as a PR model or in VR, while the shortest assessment times were recorded for the 3D PR. In addition, the majority of participants see the greatest potential for a positive influence on resection planning in the recently developed immersive VR application and prefer it over the other modalities.

3D visualizations have a high potential to enhance patient safety and surgical precision in the field of liver surgery. The meta-analysis of Zhang et al. could demonstrate that the use of 3D visualizations results in reduced intraoperative blood loss, postoperative complications, operation time, and hospitalization time, as well as a faster recovery of postoperative liver function, and lower tumor recurrence on short-term follow-up (27). In addition, Bangeas et al. conclude in their meta-analysis regarding the role of 3D printed models that 3D PR has an additional benefit over ‘just’ a 3D visualization (e.g., 3D PDF) for preoperative planning and as an educational tool. In contrast, Kong et al. found no difference between 3D PDF and 3D PR regarding anatomical education (28). However, most studies published in literature to date compare 2D CT scans with 3D PDFs either with or without an additional 3D PR (29-31). Yang et al. were the first group to compare different 3D modalities (3D PDF and 3D PR) to conventional 2D CT scans regarding tumor assignment and surgical resection planning, favoring the 3D PR reconstructions (32). The 3D VR model used in this study has first and only been published by our study group as a result of an interprofessional collaboration between surgeons and computer scientists (17). The often-used term of ‘virtual model/virtual resection’ used in other publications referrers to the standard 3D visualization on a 2D or 3D monitor, not to an immersive VR environment with VR headsets (26,33,34). This study is a first step towards an objective evaluation of currently available 3D presentation modalities factoring in newest technologies to improve anatomical education and the safety of liver surgery.

The localization of the tumor(s) within the liver is the first step towards an adequate resection planning. Therefore, the primary endpoint of the study was to determine, whether participants achieve a higher score in one of the three modalities. Statistical analyzes revealed a superiority of 3D PR models and 3D VR models over reconstructions presented as 3D PDFs. This result applies to all levels of education with the highest impact on students and the lowest impact on HPB experts. The main reason for this finding is the loss of depth perception through the projection of a 3D object on a 2D screen in the case of 3D PDFs (35). However experienced surgeons use secondary visual references to interpret non-stereo depth cues to compensate for the loss of the third dimension (36,37). The PR models can be explored using the haptic and visual sense (32). It is well investigated that especially the involvement of the haptic sense plays an important role due to the enrichment of visual perception (38). The VR model, although lacking the haptic sense input of the PR model, allows depth perception and binocular field of view, creating an immersive 3D experience (14,39). Furthermore, the ability to alter the model in size and transparency seems to compensate for the loss of haptic sense. This is supported by the correctness of tumor assignment, the results of the user experience and usability questionnaires as well as the stated subjective preferences of the participants. Thus, based on objective and subjective parameters, 3D PR models and the 3D VR models are the preferred modalities within our study population.

The time needed for the assessment of tumor-bearing segments was used as a surrogate parameter for the ability to adequately use the modality as well as the primary orientation within the 3D model. The results revealed that the 3D PR models are mostly superior to the other modalities regardless of educational level. This can be explained by the fact that the PR models are the most ‘natural’ modality, which can be explored without any technical aids. More surprisingly is the fact that with the VR application, which was unknown to the participants, assessment time was even a little shorter than with the well established 3D PDF presentation modality (40). This indicates that the new technology is easy to use and does not prolong the assessment. This thesis is also supported by the excellent evaluation results for usability (21). The quality of anatomical orientation can’t be estimated by solely analyzing the time. Nevertheless, a negative correlation between time and correctness ratio could imply a better anatomical orientation through shorter assessment time. This negative correlation can be seen for all educational levels and all modalities, not explicitly favoring one presentation format. We could show that assessment time for tumor localization is faster with 3D PR while the developed VR technology is comparable to the established 3D PDF format.

The question, which 3D modality is the best for preoperative planning, also needs to factor in production time, acquisition costs as well as availability. The segmentation of the 2D scan stands at the beginning of all 3D presentation formats. Regardless of the provider (external or local) the reconstruction is always initially presented as 3D PDF, therefore the 3D PDF is the fastest and most cost-efficient modality (15). The printing process of 3D models can take days and is associated with high costs (25,41). Also the VR application - or rather the equipment for it is costly. But in contrast to the PR models, which are invoiced per case, the procurement costs for the VR devices are only incurred once (17). Furthermore, no relevant additional time is needed to create the 3D VR model due to a drag and drop application for STL files. These factors need to be considered before using a modality for preoperative surgical planning. When it comes to education, which is the second most common application for 3D models, also sustainability and availability is of utterly importance (41). Here the breakable PR model is in clear disadvantage to the 3D PDF and the VR technology. A particular advantage of the PDF is that it can be explored on any regular computer. In addition, the PDF and the VR are not locally bound and can therefore be assessed simultaneously by different trainees in different locations. Especially the VR application enables collaborative work over distance with the potential of a multiuser mode (42). Therefore, the choice of the modality is largely dependent on the number of cases, for which a reconstruction is required, the time pressure and the application area (surgical planning/education).

A limitation of the study is a potential recognition effect when the liver model is presented the second or third time to the participant. To minimize an effect on the statistical analysis, the order of liver models as well as the order of presentation modalities was randomly selected to ensure a unique order for each participant. Furthermore, the segmentation of the liver into its segments is to some point subjective, especially if the anatomy is altered due to a malignant process. Nevertheless, surgical documentation and medical communication require the assignment of tumor-bearing segments. Therefore, we considered an expert in the field of liver segmentation and surgery (HL), to be the best way to establish a standardized correct answer.

Conclusions

This is the first preclinical study about anatomical orientation and personal preferences of three different 3D presentation modalities in liver surgery. We have shown that each modality has its advantages and disadvantages. While the PDF is most cost effective, the precision of tumor localization is higher when the reconstruction is presented as a PR model or in VR, while the shortest assessment times where recorded for the 3D PR. Nevertheless, the majority of participants preferred the VR application over the other modalities and believed in a positive influence on resection planning with this modality. Further clinical evaluation is necessary to evaluate the three presentation modalities in the context of actual preoperative, surgical planning.

The article’s supplementary files as