Augmented reality-assisted localization of solitary pulmonary nodules for precise sublobar lung resection: a preliminary study using an animal model.

BACKGROUND: Accurate localization of early lung cancer, manifested as solitary pulmonary nodules (SPNs) on computed tomography (CT), is critical in sublobar lung resection. The AR-assisted localization of SPNs was evaluated using a pig animal model.
METHODS: A Microsoft HoloLens AR system was used. First, a plastic thoracic model was used for the pilot study. Three female 12 months 45 kg Danish Landrace Pigs were then used for the animal study. Thirty natural pulmonary structures, such as lymphonodus and bifurcated bronchioles or bronchial vessels, were chosen as simulated SPNs. The average angle between the actual puncturing needle and the expected path, the average distance between the puncture point and the plan point, and the difference between the actual puncturing depth and expected depth were recorded, and the accuracy rate was calculated.
RESULTS: The point selected in the plastic thoracic model could be hit accurately with the assistance from the AR system in the pilot study. Moreover, the average angle between the actual puncturing needle and the expected path was 14.52°±6.04°. Meanwhile, the average distance between the puncture point and the expected point was 8.74±5.07 mm, and the difference between the actual and expected depths was 9.42±7.95 mm. Puncturing within a 1 cm3 area around the SPN using a hook-wire was considered a successful hit. The puncture accuracy was calculated. The average hit rate within a spherical area with a diameter of 1 cm range was 76.67%, and within a diameter of 2 cm range was 100%.
CONCLUSIONS: The HoloLens AR-assisted localization of SPNs may become a promising technique to improve the surgical treatment of early-stage lung cancer. Here, we evaluated its feasibility in an animal model. Nevertheless, its safety and effectiveness require further investigation in clinical trials. 2021 Translational Lung Cancer Research. All rights reserved.

Introduction

As is well known, lung cancer has become the malignancy with the highest rate of mortality worldwide, and is usually diagnosed at an advanced stages. Solitary pulmonary nodules (SPNs) have become increasingly detectable with improvements in high-resolution thin-slice computed tomography (CT). The nodules manifested as ground-glass opacity (GGO) on CT, with or without solid part in it. These are defined as a focal lesion with a well-defined margin and a diameter less than 3 cm (1). Accumulating evidence suggests these as primary lung cancer (2-4). Lobectomy has been the standard surgical procedure for treating lung cancer in the last few decades. However, for SPNs, especially those with GGO areas, sublobar resections, including wedge resection or segmentectomy, have been the gold standard, as reported in recent clinical trials of JCOG0802/WJOG4507L and JCOG1211 (5,6).

Precise localization before surgery has become critical for SPNs with diameters less than 1cm. In the early days, surgeons used tactile sensation to perceive and locate lung nodules during the traditional open thoracotomy. This is a feasible approach for peripheral lung nodules with more solid components, which can be identified in normal lung tissue. However, it is challenging to localize pure GGO or those much deeper in lung tissue, which have densities close to that of normal lung tissue. With the emergence of Video-assisted thoracotomy surgery (VATS), the use of a hook-wire positioning needle under CT guidance with local anesthesia before surgery has become the most common preoperative localization method. This method can accurately locate most lung nodules, except those behind the ribs or scapula, or near large vessels. Hook-wire localization under multiple CT guidance carries a risk of dislodging or uncoupling, severe pneumothorax, and air embolism (7,8). Meanwhile, since general anesthesia is not used, it can also cause immense pain to patients.

Digitization, minimally invasion, and intelligent operation are essential developments in modern surgical technology. In recent years, many innovative studies and clinical applications involving augmented reality (AR) technology have emerged in biomedicine and surgery, which are regarded as the cross-disciplinary integration of medicine and industry (9-11). AR is an innovative technology that superimpose a virtual scene into the real environment by fusing images, videos, or computer-generated models with patients during surgical operations. It can visually display the anatomical structures of organs or lesions, which significantly improves surgical efficiency.

Medical imaging technologies such as X-ray, CT, MRI can only provide a certain number of images, and cannot accurately and three-dimensionally reflect the specific anatomical position and size of the lung tumor in the human body and its adjacent relationship with the surrounding important tissue structures. CT-guided localization of SPNs presents several challenges. First and foremost is the requirement for CT equipment, special CT room or hybrid OR, and at least two radiologists, which is not easily manageable in many basic hospitals. Moreover, multiple punctures are usually needed in CT-guided localization, which is an invasive operation and brings more pain to patients in a wakeful state during the localization procedure. In addition, CT-guided localization would require multiple CT scan, thereby increasing radiation exposure on patients. Furthermore, in the cases of pneumothorax or bleeding, the transfer process or waiting process from the CT room to the operating room can be a significant risk.

Using mixed reality technology to create a new digital holographic image, the surgeons can observe from different angles, and even enter the image for observation by projecting a three-dimensional image in a real three-dimensional space. This technology can ensure smoother communication between doctors and patients and can also improve the precision of the procedure. The virtual 3D model can help the surgeon better understand the spatial location of the lesion, so as to plan the surgical path and the scope of resection. Thus, a personalized and customized surgical plan can be realized, conforming to the definition of precision surgery. Therefore, AR technology improves the success rate of the operation and effectively reduces the operation time.

In this study, AR was utilized to assist in the localization of SPNs during sublobectomy lung resection in a pig model to assess the safety and effectiveness of the technique.

We present the following article in accordance with the ARRIVE reporting checklist (available at https://dx.doi.org/10.21037/tlcr-21-554).

Methods

AR platform and modeling

Microsoft HoloLens was used to apply the AR technology in this study. The corresponding developing softwares was deployed on a PC workstation based on the x86 architecture. A Vuforia Augmented Reality developer was registered, the database was defined in the Target Manager, an AR scene was created through Unity, and the Vuforia database was imported into the scene to generate a C# project. It was then opened through Visual Studio 2017 to generate a solution that was deployed to Microsoft HoloLens.

For accurately calibrating the virtual structure on the model or pigs, what we used here is the Landmark Transform matching algorithm, which is based on the points that are designed before the operation and the points that need to be registered in a one-to-one correspondence. At the same time, it needs to meet the minimum average distance of these two point-sets. Moreover, the number of points designed before procedure and the number of the points actual need to be registered must be equal. Then the point sets with the same serial number was subjected to linear transformation for model matching.

Animal study

Experiments were performed under a project license (No. SHYS-No.202009-01) granted by institutional ethics committee of Shanghai Silversnake Biomedicine Technology Co., Ltd, in compliance with institutional guidelines of Shanghai Jiao Tong University affiliated Shanghai Chest Hospital for the care and use of animals.

A plastic thoracic model was used for the pilot study before the animal study. For this study, three female 12 months old 45 kg Danish Landrace Pigs in SPF-class were procured from Silversnake Biomedicine Technology Co., Ltd (Shanghai, China). Piggs suffering from pneumonia were excluded.

For the animal study, pigs were anesthetized with an intramuscular injection of ketamine (6 mg/kg) + diazepam (0.2 mg/kg) + scopolamine butbromide (0.3 mg), to reduce oral and airway secretions. After fixing the pig to the CT bed, a venous cannula (18−20 G) was placed into the ear vein for peripheral intravenous infusion. Then Diazepam (0.2 mg/kg), fentanyl (10 μg/kg), ketamine (2 mg/kg), and chlorosuccinylcholine (2–3 mg/kg) were injected intravenously, and the pig was intubated with a special long straight laryngoscope. Finally, the ventilator was connected, and vital signs were monitored.

A CT was performed on the model or animals, and the digital imaging and communications (DICOM) image data obtained from the scan were stored. After the first CT scan for each pig, 10 small pulmonary structures or points on the CT screen (5 each on the right and left) including calcification points, bronchiole forfication, and lymph nodes between lung segments, were chosen as assumed SPNs. A 3D Mask of the model or animal was constructed and cropped using the Crop Mask function and the Erase curve erasing tool in Edit Masks; using the Calculate Part tool to calculate the 3D model of each part; STL format files of the model were exported separately to Autodesk FBX format for the subsequent development and study; Autodesk FBX model and Vuforia Image Target database were imported into Unity to create a suitable AR scene. After compiling the data, the C# solution was generated and deployed into Microsoft HoloLens before starting the AR surgical system. Hook-wire was used for percutaneous puncture according to the AR localization of the SPNs. The CT scan and puncture were all taken at the end of inspiration by controlling tidal volume by anesthesiologist.

Data collection

During the procedure, the angles between the planned and actual puncture paths, the distance between the planned entry point and the punctured point, and the distance between the planned and actual puncture depths were all recorded and calculated. Another CT scan of the puncture result was took before the calculation for each pig. The three-dimensional position of the puncture needle was reconstructed by JinSe MIDIVI Intelligent Cloud Platform (JinSe Medical Co., Ltd. Shanghai, China). Then the three-dimensional puncture needle was compared with the preset puncture path. The main parameters for comparison are the angle between the two lines, the difference in the spatial position of the vertices of the two lines, and the distance between the actual entry point and the plan one after the puncture needle enters the body surface. Then these parameters are automatically calculated by the software Materialise 3-matic (Materialise Inc., Belgium). The hit rate of hook-wire localization was also statistically significant.

Statistical analysis

All statistical analyses were performed using SPSS 26.0 (IBM Corp., Armonk, NY, USA). The data from each experiment are presented as the mean ± standard deviation (SD).

Results

Pilot study on the plastic thoracic model

The test mainly uses two methods to complete the registration process under different dynamic and static states. After fixing the chest model on the table, the Micron Tracker probe tip was used to touch the points 1, 2, and 3 set on the model, and each point had its coordinates measured 20 times with a variance of less than 0.5 mm (). Subsequently, the researcher entered the adjusted interpupillary distance into the HoloLens system. The AR surgical navigation software was opened after wearing and adjusting the HoloLens to a comfortable state. When using the identification map to register, a researcher was required to remain still while looking at the identification map. After the virtual and real models are superimposed accurately, the OK button was clicked with gestures to fix the superimposed state.

Figure 1

A pilot study of HoloLens AR-assisted localization of point with plastic thoracic model. (A) A 3D-printed plastic thoracic model; (B) the localization under the assistance of AR was performing; (C) a blue point (in white circle) draw with marker pen was presumed as SPN, and consequently, was precisely hit without seeing it directly; (D) the view from the HoloLens during the process, and the red dot (in while circle) was in the exact position of the blue point; the green line was a plan path for puncture. This image is published with the participant’s consent.

According to the methods described above, we could determine whether the blue point drawn before the operation was hit precisely (, white circle). Meanwhile, from the view in the HoloLens, the preset points (red dots) could be clearly displayed inside of the model with a planned puncturing path and depth (). As a result, the point selected in advance could be hit accurately with the aid of the AR system, in the pilot study using the thoracic model. In the pilot study on the plastic thoracic model, three measurements of five points each were performed. Since the puncturing point and angle are fixed using blue plasticene (), only the difference between the assumed SPN (blue point in while circle in ) and the tip of the puncturing needle could be measured and analyzed. Consequently, the mean error and hit rate in a spherical area with a diameter of 1 cm range were 1.43±1.08 mm and 100% respectively ().

Table 1

The accuracy of Augmented Reality-assisted localization of targets were evaluated in plastic model

Cases	Difference between actual depth and expected depth	Mean ± standard deviation (mm)	Hit rate in 1cm
Case1	2.50 mm, 1.40 mm, 3.00 mm, 0.50 mm, 1.10 mm	1.70±1.03	100%
Case2	0.00 mm, 0.70 mm, 1.60 mm, 3.00 mm, 2.20 mm	1.50±1.19	100%
Case3	0.30 mm, 1.80 mm, 2.80 mm, 0.60 mm, 0.00 mm	1.10±1.17	100%
Total	–	1.43±1.08	100%

Pulmonary nodules can be accurately localized under the AR-assistance in pig animal study

Natural pulmonary structures, such as lymphonodus and bifurcation of bronchiole or bronchial vessels, which can be easily identified from the surrounding lung tissue on the CT screen, were chosen as SPNs in the initial CT images of pigs (, the red dot indicates the SPN selected). Once the point was selected, it was processed for 3D-reconstruction using Materialise Mimics V17.0, and the puncturing path and depth were planned ().

Figure 2

The reconstruction of SPNs and radiological analysis. (A-C) Coronal, transversal, and sagittal images of SPN (Red dot); (D) simulation and 3D reconstruction of the SPN and plan puncturing path (Red line).

After the pigs were anesthetized on the CT bed, the AR HoloLens was put on and the preset SPNs were displayed on the surface of the pig’s chest from the researcher’s view (). A hook-wire localizing needle was used to puncture the area according to the specific position and angle in the AR view to the recorded depth (). When all points were punctured and localized, another CT was performed to check the results, and it was found that the SPN was accurately hit (). All the results were recorded and analyzed ().

Figure 3

The procedure of localization of SPNs in pig animal. (A,B) The researcher adjusted the interpupillary distance in the HoloLens system. After the HoloLens was worn and adjusted to a comfortable state, the AR surgical navigation software was opened; (C) the preset SPNs were processed for 3D-reconstruction using Materialise Mimics V17.0, and the puncturing path and depth were planned; (D-F) SPNs were punctured using hook-wire under the aid of AR planning position and path. This image is published with the participant’s consent.

Figure 4

Representative radiological images of localization of SPN. (A) a SPN was selected and marked as red dot in red circle before puncture; (B) this SPN was accurately hit using hook-wire.

Table 2

The accuracy of Augmented Reality-assisted localization of solitary pulmonary nodules were evaluated in pig animal study.

Cases	The angle between puncturing line and expected path (°)	Distance between actual entry point and expected point (mm)	Difference between actual depth and expected depth (mm)	Hit rate in 1 cm	Hit rate in 2 cm
Case 1
Right	16.12±7.32	8.74±7.65	14.56±9.23	80%	100%
Left	16.95±5.02	10.26±2.57	5.95±1.79	100%	100%
Case 2
Right	8.8±3.74	6±2.11	4.94±4.47	100%	100%
Left	7.53±6.28	3.75±3.95	9.05±6.97	80%	100%
Case 3
Right	13.08±6.70	7.97±7.92	15.85±11.2	50%	100%
Left	14.74±5.23	5.76±4.34	18.62±9.80	50%	100%
Total	14.52±6.04	8.74±5.07	9.42±7.95	76.67%	100%

Consequently, 30 selected SPNs from three pigs were punctured using hook-wire and AR-assisted HoloLens for localization. The average angle between the actual puncturing needle and the expected path was 14.52°±6.04°. Meanwhile, the average distance between the puncture point and the expected point was 8.74±5.07 mm and the distance between actual depth and expected depth was 9.42±7.95 mm. Puncturing within a 1 cm area around the SPN using a hook-wire was considered a successful hit. The accuracy of the puncture was calculated, and the average hit rate in a spherical area with a diameter of 1 cm was 76.67%. However, the hit rate in a spherical area with a diameter of 2 cm was 100% ().

Discussion

Human society is moving from the information and digital age to the intelligent age. Digital, minimally invasive, and intelligent surgery are important directions for modern surgical technology, and are important signs of advanced surgical technology. AR technology, also known as augmented or virtual reality, allows the virtual world on the display system to interact and be combined with the real world scene, through the accurate calculation of the position and angle of the camera video and the image analytical technology (12). AR usually includes three aspects: combining virtual objects with reality, instant interaction, and three-dimensional imaging. This technology can be distinguished from VR. When AR is applied, the user's eyes can see not only the real world but also the virtual world derived from the computer through objects and the environment from the real world. The use of image guidance and image-based surgical navigation technology can ensure the accuracy and safety of the operation (13). When the surgeon’s eyes are looking at the screen of the surgical navigation system and the hands are operating in the surgical area, the accuracy of the operation is easily jeopardized due to improper hand-eye coordination, ultimately reducing the efficiency of the operation (14).

According to previous reports, the many methods for localization of SPNs can be categorized as pre- and mid-operation ones (15) (). The mid-operation methods include palpation by the surgeons and ultrasound. Apparently, these two methods possess a high error rate. As for ultrasound localization, the lesions are limited to subpleural areas less than 2 cm in depth, and the lung tissue needs to be collapsed entirely because, in the poorly collapsed state, interference from the residual gases can lead to failure of the procedure. In addition, some factors that may affect the localization of lung collapse, such as diffuse emphysema or bullae, chronic obstructive pulmonary disease, and pulmonary fibrosis, may affect the collapse of the lung, causing failure of ultrasound localization (16,17).

Table 3

Advantages and disadvantages of most common used localization methods of solitary pulmonary nodules

Drawbacks	Application in VATS	Error rate	Complications	Pain	Short maintenance time	Blind locating area	High requirement of apparatus	Expenditure	CT Radiation	Transferring risk
Mid-operation	Almost been abandoned
Palpation		●
Ultrasound		●					●
Pre-operation	Currently used
Hook-wire localization under CT	●		●	●			●	●	●	●
Radioactive iodine oil	●		●	●		●	●	●	●	●
Methylene blue mapping	●	●	●	●	●		●	●	●	●
Magnetic navigation guidance	●		●			●	●	●
AR-assisted localization	●

Black dot indicates conform to the situation.

The preoperative localization methods include hook-wire puncturing localization under CT scanning, radioactive iodine oil, methylene blue mapping, magnetic navigation guidance, and AR-assisted localization. Currently, hook-wire puncture localization under CT scan is the most commonly used method for the preoperative localization of small pulmonary nodules. However, since the hook-wire needle is so sharp and barbed, it can readily give rise to complications such as intrapulmonary hemorrhage, hemoptysis, chest wall hematoma, hemopneumothorax, and even air embolism, if it is accidentally uncoupled. Meanwhile, hook-wire localization is affected by respiratory and body position, leading to failure in localization. In 1992, Kerrigan et al., first used the localization method to inject methylene blue near pulmonary nodules (18). Yao et al., proved the safety and feasibility of this method (19). Methylene blue localization has the advantages of low price and reduced discomfort to patients. However, many studies have reported that methylene blue localization alone can cause various complications, such as pneumothorax, hemothorax, embolism, and fatal allergic reaction (20). Additionally, methylene blue also diffuses easily, reducing the accuracy of localization of SPNs (21). Electromagnetic navigation bronchoscopy (ENB) technology was first used in clinical practice in the United States (22). CT and computer software systems were used in ENB technology to reconstruct the virtual image, mark the target lesion, and plan the insertion path. The advantages of this method are that it can reach the parts that cannot be reached by conventional bronchoscopy or lung puncture. In addition, it is safe, fast, and less invasive. The success rate of biopsy can be as high as 91.8%, as reported by Khandhar et al. (23). However, this method has high requirements for equipment and personnel, is time-consuming, and has a long operation time.

In recent years, head-mounted AR or VR equipment and various mature commercial products have been widely used in the field of surgery. Microsoft HoloLens is relatively simple and portable and can be applied to more scenarios. During the operation, AR technology can not only project the fluoroscopy images of the operation area to the surgeon in real-time, but also display the patient’s anatomy as a three-dimensional holographic image. Surgeons’ experience will be more intuitive and specific, which will significantly improve their learning curve. The use of AR technology can avoid the disturbance of hand-eye coordination during surgery. The AR display system can simultaneously present objects in the real world and elements in the virtual world. The operator's eyes can remain fixated on the surgical area, with the relevant information displayed in the corresponding position of the surgical area. In particular, when head-mounted AR devices are used, the operator can clearly see the actual surgical area through the screen because some head-mounted AR devices use semi-transparent screens, thus avoiding possible problems caused by hand-eye separation.

In the present study, HoloLens AR-assisted localization of SPNs was evaluated in a pig animal model. The safety of this technology has been well demonstrated since the vital signs of experimental pigs were all normal during the whole study. Furthermore, according to our results, the average distance of the actual puncturing entry point and plan point was 8.74±5.07 mm, and the average distance of puncturing depth and expected depth was 9.42±7.95 mm, both of which were less than 1 cm distance. The total rate of accuracy was 76.67% when the hit range was set to 1 cm in diameter around the nodule and 100% when it was set to 2 cm in diameter around the lesion, which is acceptable in pulmonary segmentectomy or wedge lung resection.

HoloLens is the most suitable AR display device for surgical navigation. Qian et al. compared the system delay time and frame rate of the three AR devices, including EpsonBT200, ODGR7, and HoloLens, through experiments in a simulated surgical environment (24). The results showed that HoloLens was superior to the other devices. Philip et al. applied HoloLens to leg perforator flap surgery using the patient’s CT image data for three-dimensional reconstruction to obtain virtual anatomical models such as blood vessels and skin (25). During the operation, the virtual model was superimposed and registered to the corresponding anatomical structure of the patient, and HoloLens was used for three-dimensional display. Incekara et al., applied HoloLens to 25 cases of brain tumor resection surgery, and the average margin of error was 4 mm, as measured with the traditional neurosurgery navigation system, BrainLab, as the standard control (26). Perkins et al. reported that Mixed-reality visualization during surgical planning facilitate accurate and rapid identification of small lung lesions during minimally invasive surgeries and reduce the need for additional invasive preoperative localization procedures using Microsoft HoloLens as well (27).

Safety is an important indicator of this kind of new technology. Due to the proof-of-concept animal experiment in this laboratory, the experimental animals are under general anesthesia and vital signs are monitored during the entire experiment. We did not observe obvious adverse reactions such as hemothorax and cardiovascular events. There was only one pig (case 3 in ) manifested mild pneumothorax, because of which the hit rate in 1 cm was just 50% in this case, and closed thoracic drainage was performed in time. Further exploration and verifying about the safety of this method was expected in the follow-up phase I clinical study.

Compared with the localization of SPNs under CT guidance, AR-assisted localization technology apparently has many advantages (). It is less time-consuming and can be performed immediately before surgery under general anesthesia, lessening pain. Moreover, AR-assisted localization does not require a CT room or a hybrid OR because the HoloLens is portable. In addition, it excludes the risk of radiation damage caused by CT. Nevertheless, the limitations of AR-assisted localization technology are that it requires specific software and technical personnel, and the pre-clinical study requires optimization to enhance the rate of accuracy. As for the limitations of this study, there is a learning curve of AR-assisted localization of SPNs in pigs, which can lead to unexpected results in the early stage. Enrolling more pigs and conducting more clinical trials would be needed to better understand the potential of this technology.

Table 4

The comparison of localization of solitary pulmonary nodules under the CT guidance and augmented reality assistance

	Localization under CT guidance	Localization under augmented reality
Time	20–30 min	5–10 min
Radiation	Repeated radiation	No radiation
Waiting time	0–2 hours	0 hours
Pain	Local anesthesia, intense pain	General anesthesia before surgery
Requirement on apparatus	CT, immovable	HoloLens, portable
Requirement on circumstance	CT room	Non, operation room
Expenditure	CT maintenance cost, localization cost for 2–3 staffs	Model reconstruction cost

In conclusion, the HoloLens AR-assisted localization of SPNs may become a promising technique to improve the surgical treatment of early stage lung cancer. Here, we evaluated its feasibility in an animal model. Nevertheless, ensuring its safety and effectiveness would require further investigation in clinical trials.

The article’s supplementary files as