Virtual reality three-dimensional echocardiographic imaging for planning surgical atrioventricular valve repair.

OBJECTIVES: To investigate how virtual reality (VR) imaging impacts decision-making in atrioventricular valve surgery.
METHODS: This was a single-center retrospective study involving 15 children and adolescents, median age 6 years (range, 0.33-16) requiring surgical repair of the atrioventricular valves between the years 2016 and 2019. The patients' preoperative 3-dimesnional (3D) echocardiographic data were used to create 3D visualization in a VR application. Five pediatric cardiothoracic surgeons completed a questionnaire formulated to compare their surgical decisions regarding the cases after reviewing conventionally presented 2-dimesnional and 3D echocardiographic images and again after visualization of 3D echocardiograms using the VR platform. Finally, intraoperative findings were shared with surgeons to confirm assessment of the pathology.
RESULTS: In 67% of cases presented with VR, surgeons reported having "more" or "much more" confidence in their understanding of each patient's pathology and their surgical approach. In all but one case, surgeons were at least as confident after reviewing the VR compared with standard imaging. The case where surgeons reported to be least confident on VR had the worst technical quality of data used. After viewing patient cases on VR, surgeons reported that they would have made minor modifications to surgical approach in 53% and major modifications in 7% of cases.
CONCLUSIONS: The main impact of viewing imaging on VR is the improved clarity of the anatomical structures. Surgeons reported that this would have impacted the surgical approach in the majority of cases. Poor-quality 3D echocardiographic data were associated with a negative impact of VR visualization; thus. quality assessment of imaging is necessary before projecting in a VR format.

Virtual reality imaging for 3D echocardiography in use.

Virtual reality dynamic 3-dimensional echocardiographic imaging improves surgical insight for atrioventricular valve repair planning in congenital heart disease for clinical use.

This study demonstrates the potential clinical benefits and value of virtual reality in surgical planning for congenital heart disease and other structural heart defects. The observed benefits are improved user interaction and visualization of valve apparatus in a beating heart compared with image visualization using standard techniques.

See Commentary on page 278.

Pediatric and congenital disease of atrioventricular (AV) valves presents a surgical challenge due to the large amount of anatomical variability between individual patients, affecting the nature and extent of the defect requiring repair. Valve replacement is a suboptimal outcome in growing patients because serial valve replacement is inevitable with growth. Surgeons rely on imaging modalities such as 2-dimesnional (2D) and 3-dimesnional (3D) echocardiography to provide real-time dynamic imaging of AV valves, supplemented by computed tomography (CT) and magnetic resonance imaging. A major limitation of these imaging techniques is that they project a 3D structure on a 2D screen, with limited depth perception.

3D-printed models offer true depth perception of the heart and can be cut, retracted, and sutured., However, these models are static and either do not include AV valves at all or only a static representation. Virtual reality (VR) environments are an attractive alternative for patient-specific image visualization. They have the advantage of enabling true depth perception and being able to display moving 3D images, which is critical for preprocedural planning of repair of AV valves. However, the true clinical benefit of VR on improving patient outcomes remains poorly demonstrated.

In this study, a prototype for such an immersive VR system was created in-house (www.3dheart.co.uk) via the integration of Unity and the Visualization Toolkit, which are, respectively, a 2D/3D/xR video game developing platform and a visualization library specifically designed for medical imaging and implemented on an HTC Vive VR Headset. Development of this system was supported by a National Institute for Health Research Invention for Innovation Grant (II-LA-0716-20001). The aim of the study was to critically evaluate the potential impact of presentation of 3D echocardiographic data, in addition to conventional 2D/3D echocardiograms, in patients who had undergone such surgery at our center during a 3-year period. We aimed to investigate whether VR imaging impacted surgeons’ perception of the anatomy and whether this might have impacted on their surgical approach.

Study Design

Children undergoing surgical repair of AV valves at the Evelina London Children's Hospital between July 2016 and October 2019 were screened for inclusion. Twenty-eight consecutive patients were identified as having undergone AV valve surgery, with 10 cases excluded due to the lack of any or adequate 3D echocardiographic data. Patients undergoing primary repair of complete AV septal defects were also excluded to focus the area of study to repair AV valves alone. Patients with adequate 3D transesophageal (TOE) or transthoracic (TTE) images were both included. 3D TOE at the time of study was generally only feasible in patients >25 kg. Datasets from 15 patients with a median age of 6 years and 3 months, and a range of 4 months to 16 years and 2 months were included as case studies.

The electronic patient record was accessed to obtain relevant clinical data including the operative notes, multidisciplinary meeting minutes, and discharge letters. Each patient's clinical data were summarized and anonymized to form standardized case vignettes to present to surgeons individually in a simulated preoperative setting. This was similar to our normal approach before surgery, except that surgeons were asked individually, rather than as a group, to avoid bias. The patients' relevant preoperative 2D and 3D ultrasound scans were selected and anonymized for review by participating surgeons. The corresponding 3D datasets were also anonymized and exported to the VR system as shown in Figure 1 (Video 1). Every case had anatomic landmarks labeled and orientation adjusted to the presenting view by a cardiologist (K.P., J.M.S.) to show images in an anatomic formal in keeping with published guidelines and our usual practice for consistency. The labeled landmarks are retained during image manipulation as a feature of the system. The random allocation of cases to surgeons was achieved by giving each surgeon a letter from A to E and each patient case a number from 1 to 15, then pairing them randomly via an online randomization tool. This process was repeated until each participating surgeon was assigned 3 cases that they had not operated on to avoid bias from previous insight on the case.

Figure 1

Virtual reality image display of a 3-dimensional transesophageal echocardiogram image as viewed from the atrial aspect demonstrating a perforation of the anterior mitral valve leaflet (Video 1). The user interacting with the image is seen in the inset image. A handheld device enables interactive control of the view, slicing plane, and imaging properties. MV, Mitral valve; TV, tricuspid valve.

A questionnaire was formulated to collect feedback from the surgeons about their surgical opinions on the patient cases, which accompanied the imaging. A brief description of the questionnaire sections is outlined below, whereas the full questionnaire is available for review in Online Data Supplement.

Part 1 gathered information about each surgeon including their experience;

Part 2 was completed by the surgeon after viewing a patient's 2D and 3D echocardiographic images on conventional software (QLAB; Philips, Andover, Mass);

Part 3 was completed by surgeons following use of the VR system, including questions similar to those in Part 2 but focused more on the changes, if there were any, in their confidence on the pathology and whether they would modify their surgical approach;

Part 4 sought to compare the operative surgical findings to the surgeon's understanding of the anatomy having reviewed 2D, 3D, and VR imaging of the patient; and

Part 5 enquired about the surgeons' general views on the potential for VR to be used in surgical planning

The data-collection process involved a 1-hour appointment with every surgeon to view 3 patient cases each. These steps are summarized as a flowchart (Figure 2). They first completed Part 1 of the questionnaire, then viewed their first allocated patient summary vignette and standard imaging scans, and subsequently completed Part 2 of the questionnaire. They viewed the same patient's images on VR and completed Part 3. Then they were shown the case's actual operative findings and completed Part 4. After this process, parts 2 to 4 were repeated for their second and third patients. When all patient data had been viewed, Part 5 of the questionnaire was completed.

Figure 2

Flow chart of surgical imaging study using standard echocardiographic imaging and virtual reality. VR, Virtual reality.

Institutional ethics approval was obtained permitting retrospective review and use of echocardiographic data for clinical research and publication (09/H0802/116) without the need for informed written consent for publication.

Results

Surgical Participant Information

Five pediatric cardiothoracic surgeons were recruited to participate in the study. Three of them were senior surgeons of more than 15 years' experience, one was a senior trainee with more than 3 years' experience, and one was a junior trainee of less than 3 years’ experience. Their ages ranged from 31 to 57 years. Three surgeons reported having used VR once or twice in the past, whereas the other 2 had never used VR before and were given a standardized pre-prepared short tutorial with defined tasks on VR equipment usage before their viewing of the scans using the VR system.

Imaging Data

Patients in presented cases had a median age of 6.25 years (0.33-16.16 years). The median patient weight was 12.5 kg and ranged from 4.4 to 70 kg. Types of AV valve lesions included atrioventricular septal defect, mitral valve stenosis, regurgitation, and prolapse. Four patients underwent TOE imaging and the remaining 9 underwent TTE imaging.

Impact of VR on Understanding of Pathology

After viewing the patients' conventional 2D and 3D echocardiograms, the median initial confidence of the surgeons was 4 of 5 (range, 2-5) with regard to their understanding of the patients' pathology and 4 (range, 2-4) with respect to their preferred surgical approach. The bar charts below display the surgeons' change in confidence regarding the patients’ pathology requiring surgical attention (Figure 3) and their surgical approach to repairing the valve (Figure 3) after viewing the images on the VR platform compared to conventional 2D and 3D echocardiography. In 10 of 15 cases, the surgeons felt “more confident” or “much more confident” about the pathology requiring surgical intervention (Figure 3). An identical proportion felt “more confident” or “much more confident” in their potential surgical approach (Figure 4). The individual surgical responses are displayed in Table 1. The following direct quotations from some of the responses that surgeons provided to the question “If there is a better understanding of the condition, please describe the specific areas where this was helpful” that illustrate how VR was useful to them:

Figure 3

Change in surgeon confidence regarding patient pathology after viewing in virtual reality compared with assessment after standard imaging. VR, Virtual reality.

Figure 4

Change in surgeon confidence regarding surgical approach after viewing in virtual reality compared with assessment after standard imaging. VR, Virtual reality.

Table 1

Surgical comments on visualization using virtual reality compared with standard 2D and 3D echocardiography

Visualization of the lesion much clearer.

Probably with more complex lesions that are harder to evaluate from 2D and 3D, VR might offer some more understanding; however, with the given case, I thought I've had good understanding from the beginning, so I feel VR only offered more amusement to my experience.

How little the posterior leaflet moves.

Everything just looks clearer and I can manipulate the orientation to my surgical perspective in all directions.

The leaflets looked worse on the VR simulation than what I could understand from the echo. Chord lengthening wouldn't be helpful in this scenario and posterior leaflet patch wouldn't make such a difference. The valve looked maybe too dysplastic for repair, but I'd give a try.

Able to visualize the precise leaflet component that is prolapsing but not able to see the reason, ie, is it chordal rupture?

There is a bigger area of noncoaptation in A1, which I had felt was A2.

I could understand better the leaflet disposition with the VR rather than the echo.

Better visualization of the precise location of the defect.

VR gave me a clearer image and understanding of the LAVV in this particular patient.

Yes. How the different scallops of the MV leaflet move in relation to the cardiac cycle. A2 and P2 close (and open) later.

I could see better the absence of coaptation of both leaflets and the annular dilatation. Prolapse in A1-P1 seemed to me more prominent in VR. Posterior leaflet looked tethered in VR.

2D, 2-Dimensional; 3D, 3-dimensional; VR, virtual reality; LAVV, left atrioventricular valve; MV, mitral valve.

Visualization of the lesion much clearer.

I could see better the absence of coaptation of both leaflets and the annular dilatation. Prolapse in A1-P1 seemed to me more prominent in VR. Posterior leaflet looked tethered in VR.

I had better visualization of the precise location of the defect.

I could understand better the leaflet disposition with the VR rather than the echocardiogram.

Everything just looks clearer, and I can manipulate the orientation to my surgical perspective in all directions.

Impact of VR on Surgical Modifications

When asked whether they would modify their surgical approach after viewing the images using the VR system, surgeons chose to make “minor modifications” in 8 of the 15 cases (53%), “major modifications” in 1 case (7%), and “no modifications” (40%) for the remaining 6 cases. The individual responses detailing major and minor modifications are listed in Table 2.

Table 2

Surgical comments on modifications to surgical approach following visualization using virtual reality compared with standard 2D and 3D echocardiography

Major modifications

Try to set free the edges of both leaflets by secondary chord resection and commissuroplasty.

Minor modifications

I think a patch augmentation of the posterior leaflet can increase the coaptation.

There needs to be fenestration of the posterior chordal arcade, along with division of secondary chordae patch still very likely.

There is a bigger deficiency in the area of A1/P1 than I appreciated on the 2D/3D, so I would concentrate more on this area but still free up the posterior leaflet, as I would suture more of A1 to P1 with an annuloplasty in this area.

Closing the cleft, bringing together the left mural and superior bridging leaflet.

Looking at VR made me more confident with the need for posterior annuloplasty and possible papillary muscle commissuroplasty.

A2 chord replacement (shortening), ring annuloplasty +/– cord replacement on P2.

I do not think the supramitral membrane is so visible and, if so, only on the posterior leaflet. The subvalvular apparatus still needs fenestration.

Probably would free as well the edge of the posterior leaflet, as it seemed to me that it was tethering a bit. There's a big absence of coaptation also in A1-P1.

2D, 2-Dimensional; 3D, 3-dimensional; VR, virtual reality.

The surgeon reporting the decision to make “major modifications” after viewing VR imaging originally would have done “cord lengthening and patch extension of posterior leaflet” based on just viewing standard imaging, whereas after viewing the scans on VR, their surgical approach changed to “try to set free the edges of both leaflets by secondary chord resection and commissuroplasty.”

This surgeon cited better understanding of the condition specifically in the following manner:

“The leaflets looked worse on the VR simulation than what I could understand from the echo. Chord lengthening wouldn't be helpful in this scenario and posterior leaflet patch wouldn't make such a difference. The valve looked maybe too dysplastic for repair, but I'd give a try”

Notably, in one patient case, a surgeon rated “no change” in confidence regarding both the patient pathology and the surgical approach after viewing the VR system. The surgeon's comments on are as follows:

“Probably with more complex lesions that are harder to evaluate from 2D and 3D, VR might offer some more understanding, however, with the given case, I thought I've had good understanding from the beginning, so I feel VR only offered more amusement to my experience.”

The one case in which the surgeon reported being “much less confident” in the patient's pathology and their surgical approach commented that “looking at VR in this particular case didn't add any more information and therefore didn't change my approach” and that the VR “didn't really give me any better understanding in this particular case.”

Comparison With In Vivo Surgical Findings

The surgeons were asked to compare their assessment of the patients' condition with those of the operating surgeon recorded in the surgical postoperative notes. The surgical findings were consistent with the assessments from the VR study in 11 of 15 patients. In 4 cases, the surgeons reported that their own evaluation did not match the surgical findings. The reasons for these inconsistencies included an underestimated degree of prolapse, misjudged role of the membrane, unappreciated annulus dilatation, and a missed commissural stitch. Descriptions of these discrepancies in surgical finding are displayed in the surgeons’ own words and level of experience (Table 3).

Table 3

Descriptions of discrepancies in surgical findings in the surgeons’ own words and level of experience

Level of experience	Comments
Senior independent practitioner, more than 5 y	I underestimated the degree of prolapse from both 2D and 3D, but this would not have changed the surgical approach or intraoperative evaluation.
Senior independent practitioner, more than 5 y	It was more or less what I thought of after the VR assessment. However, I couldn't appreciate much the annulus dilatation and that's why I didn't think of the annuloplasty. Probably is more due to my lack of experience rather than not being seen on VR.
Senior independent practitioner, more than 5 y	The membrane was obviously a bigger player but then the surgeon had to do further subvalvular surgery with papillary muscle splitting, which I felt did need to be done from the images.
Junior trainee, less than 3 y	I thought of implanting neochordae and annuloplasty. There was also a commissural stitch between A3 and P3 that I didn't think of. Probably I didn't assess this segment properly neither on echo nor in VR, but roughly, the approach was quite similar to what I initially thought of.

2D, 2-Dimensional; 3D, 3-dimensional; VR, virtual reality.

Surgeons’ View of VR in Future of Surgery in General

The final section of the questionnaire—regarding the surgeons' general view on the VR system in the future of surgery—yielded the unanimous results from both surgeons in all fields. All agreed that the VR system could help decrease surgical procedure time and become a routine clinical tool for planning congenital cardiac surgery, whereas 3 of 5 agreed that the VR system could help diminish potential surgical complications. A summary of the methodology and results is outlined in Figure 5.

Figure 5

This summarizes the methodology, key results and implications of virtual reality 3D echocardiographic imaging for planning surgical atrioventricular valve repair. Fifteen pediatric atrioventricular valve repair cases were assessed by 5 surgeons using standard 2D/3D ultrasound, and again using a virtual reality platform. Virtual reality was shown to improve surgical confidence in understanding the relevant anatomy for repair, resulting in modifications to their proposed surgical approach. The implications of this are also outlined. AV, Atrioventricular; 2D, 2-dimensional; 3D, 3-dimensional; USS, ultrasound; VR, virtual reality; CHD, congential heart disease.

Discussion

Advances in computerization and software development mean that it is now feasible to project 3D echocardiographic images into a VR environment similar to cardiac CT and magnetic resonance imaging data. A central question, however, is whether such technology might improve the approach to surgery or intervention and patient outcomes. The ideal approach would be to study patients prospectively before surgical intervention. However, this raises ethical challenges in the absence of regulatory approvals, for example, CE (Conformitè Europëenne) marking, for prospective use of the VR system, which may bias surgical approach in a prospective study. Thus, the approach we adopted was to study echocardiographic images on patients who had already been operated to gauge whether VR might have influenced surgical confidence in details of the lesion and the approach to surgical repair. Our selection of cases who underwent repair of AV valves was to test the technology in a clinically relevant and challenging surgical situation.

We have demonstrated the clinical usefulness of VR in consecutive patients undergoing repair of AV valves, selected only on the basis of available 3D echocardiographic data, which is in routine use in our department. Our data show that in the majority of cases VR improved the surgeon's understanding of the valve to be repaired and had either a minor or major influence on the surgical approach. Specifically, they described the VR to have given them better visualization of anatomical structures, leaflet motion, and the opportunity to manipulate image orientation for surgical perspectives. The results show that the main benefit of VR visualization was the detail of anatomy and customizable views of the valve motion. This contrasts with 3D prints, which cannot permit such interactive, manipulable, and dynamic assessment of AV valves. 3D visualization from static 3D printed models has already been shown to improve understanding of anatomy and helped redefine the surgical approach in up to 47% of cases with complex congenital heart disease with significant modifications in 25% of cases. However, in that study, the precise contribution of the 3D model in the change of surgical thinking was not explored as we have done here. The VR system also allows intuitive interaction so that the surgeon has the freedom to manipulate and orientate images to their surgical perspective in all directions. Compared with 3D-printed models, VR is inexpensive, reproducible and, importantly, swiftly available at the point of care. Even an experienced surgeon gaining confirmation rather than new information may have more confidence in dealing with the problem.

The discrepancies in comparing VR with surgical operative findings were only seen in 4 cases. One case reported an underestimation of the degree of prolapse. The prolapse was not a new finding and was compared with assessment in theater in a nonbeating heart with saline testing by a different operator thus introducing a variability in assessment methods. The 3 other cases included underestimation of the degree of annular dilation, severity of a supramitral membrane, and need for a stitch between A3 and P3 scallops. With respect to annular dilatation, our VR system has the capability to make measurements, but this was not evaluated as part of the study. Supramitral membranes are recognized to be challenging to image by echocardiography and may reflect the challenges of input data rather than VR portrayal. Moreover, in all these cases, the surgeons describe subtle observations that may be identified in the datasets on VR with increased use and familiarity of features VR systems can offer. In this study, we used the ability of the prototype system to label and landmark images to aid with image orientation during manipulation by surgeons for every case in this study. This is a feature in the VR system that we introduced during the development phase after we identified it as a key need to end-user interaction and acceptability for surgical planning and is reflected in other studies in this area. Napa and colleagues compared 2 commercially available VR systems for assessment of cardiac features and highlighted the importance of user-friendly and acceptable means of labeling and user interaction. There are already published image orientation guidelines for 3D echocardiography that can be immediately incorporated into VR developments to maintain a standardized approach in this emerging field. Artificial intelligence tools can be incorporated into VR modalities to learn a surgeons' preferred view adjustments and most commonly used settings to create convenient defaults in a VR program. The combination of VR and artificial intelligence can thus be used to reorient images and automatically present the patient's cardiac anatomy in a standardized orientation for surgeons to view systematically and manipulate as they choose from that point.

We identified 1 example in which use of VR reduced the confidence of the surgeon in their surgical findings and approach. Further review of the case noted that poor input data quality played a likely role in the discrepancy from other cases where VR use was not detrimental. The patient was an unsedated baby with a high respiratory and heart rate, all of which are known to affect the quality of 3D TTE acquisitions. Hence, the added benefit of VR imaging is dependent on the quality of the 3D data input. However, 3D echocardiography is a rapidly evolving field with increasingly refined algorithms making it easier and faster to produce good 3D datasets on commercially available systems.

Carnahan and colleagues have successfully employed multiview 3D echocardiography volume compounding to achieve full coverage of mitral valve, cordae, and papillary muscles for surgical planning in mitral valve surgery. In a separate publication, compounded data of this type have been successfully viewed on our VR platform allowing for interrogation of such complex datasets and assessment of the mitral valve cords. This paves the way for more detailed planning in cases needing cordal intervention such as placement of artificial cords as part of the AV valve repair. The use of compounded 3D echocardiography data in VR has significant potential in this field as most 3D printing studies of the AV valve to date are limited to CT data for its high spatial resolution and wide coverage despite being static, while those using standard 3D echocardiography are limited to study of the valve leaflets alone.

There are case reports using virtual reality in surgical planning, demonstrating some of the benefits and interest in this field for adoption into congenital heart disease and other forms of structural heart disease and planning placements of ventricular assist devices. Importantly, the ability to share image visualization is key to promoting wider discussion, as is usually needed in planning complex cases. Structured evaluations with validation of these techniques will help inform the wider uptake of these methods into standard of care.

Limitations

Given the varied levels of experience of the participating surgeons using VR technology, it is possible that despite the brief introduction to VR included in this study, there could be a learning curve effect on the observed results. However, it is likely that the added value of VR will only increase with more experience and familiarity with the system and we were still able to demonstrate a significant advantage from using the VR system. In addition, a crossover component where cases allocated to senior surgeons would be assessed again by junior surgeons, and vice versa would investigate the contribution of VR to surgeons of different levels of experience in recognizing cardiac pathology and formulating surgical approaches and help specify the role of VR imaging for different users. The sample size was too small to assess the effects of previous surgical experience on the observed benefits of VR. The VR system can make cardiac measurements, but this was not evaluated as part of this study.

Conclusions

VR has been shown to offer a clear benefit in surgical understanding, confidence, and planning for surgical repair of AV valves in congenital heart disease over conventional 2D and 3D echocardiographic imaging. The gains are from an enhanced visualization of subtle anatomical detail and the freedom of the surgeon to interrogate the dynamic images of the valve structures themselves.

Conflict of Interest Statement

The authors reported no conflicts of interest.

The Journal policy requires editors and reviewers to disclose conflicts of interest and to decline handling or reviewing manuscripts for which they may have a conflict of interest. The editors and reviewers of this article have no conflicts of interest.