Junjun Pan1, Weimin Liu2, Pu Ge2, Fanghong Li3, Weiyun Shi4, Liyun Jia5, Hong Qin6. 1. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China; Peng Cheng Lab, Shenzhen, China; Faculty of Media and Communication, Bournemouth University, Bournemouth, UK. Electronic address: pan_junjun@buaa.edu.cn. 2. State Key Laboratory of Virtual Reality Technology and Systems, Beihang University, Beijing, China. 3. Shenzhen Kechuang GuangTai Technology Co.,Ltd., Shenzhen, China. 4. Shandong Eye Institute Shandong Eye Hospital, Jinan, China. 5. Beijing Tongren Eye Center, Beijing Tongren Hospital, Capital Medical University, Beijing Ophthalmology and Visual Sciences Key Laboratory, Beijing, China. 6. Department of Computer Science, Stony Brook University, New York, US. Electronic address: qin@cs.stonybrook.edu.
Abstract
OBJECTIVE: Corneal disease is one of the main causes of blindness for humans globally nowadays, and deep anterior lamellar keratoplasty (DALK) is a widely applied technique for corneal transplantation. However, the position of stitch points highly influences the success rate of such surgery, which would require accurate control and manipulation of surgical instruments. METHODS: In this paper, we present a deep learning framework for augmented reality (AR) based surgery navigation to guide the suturing in DALK. It can robustly track the excised corneal contour by semantic segmentation and the reconstruction of occlusion. We propose a novel optical flow inpainting network to recover the missing motion caused by occlusion. The occluded regions are detected by weakly supervised segmentation of surgical instruments and reconstructed by key frame warping along the completed optical flow. Then we introduce two types of loss function to adapt the inpainting network in the optical flow space. RESULTS: Our techniques are tested and evaluated by a number of real surgery videos from Shandong Eye Hospital in China. We compare our approaches with other typical methods in the corneal contour segmentation, optical flow inpainting and occlusion regions reconstruction. The tracking accuracy reachs 99.2% in average and PSNR reaches 25.52 for the reconstruction of the occluded frames. CONCLUSION: From the experimental evaluations and user study, both the qualitative and quantitative results indicate that our techniques can achieve accurate detection and tracking of corneal contour under complex disturbance in real-time surgical scenes. Our prototype AR navigation system would be highly useful in clinical practice.
OBJECTIVE:Corneal disease is one of the main causes of blindness for humans globally nowadays, and deep anterior lamellar keratoplasty (DALK) is a widely applied technique for corneal transplantation. However, the position of stitch points highly influences the success rate of such surgery, which would require accurate control and manipulation of surgical instruments. METHODS: In this paper, we present a deep learning framework for augmented reality (AR) based surgery navigation to guide the suturing in DALK. It can robustly track the excised corneal contour by semantic segmentation and the reconstruction of occlusion. We propose a novel optical flow inpainting network to recover the missing motion caused by occlusion. The occluded regions are detected by weakly supervised segmentation of surgical instruments and reconstructed by key frame warping along the completed optical flow. Then we introduce two types of loss function to adapt the inpainting network in the optical flow space. RESULTS: Our techniques are tested and evaluated by a number of real surgery videos from Shandong Eye Hospital in China. We compare our approaches with other typical methods in the corneal contour segmentation, optical flow inpainting and occlusion regions reconstruction. The tracking accuracy reachs 99.2% in average and PSNR reaches 25.52 for the reconstruction of the occluded frames. CONCLUSION: From the experimental evaluations and user study, both the qualitative and quantitative results indicate that our techniques can achieve accurate detection and tracking of corneal contour under complex disturbance in real-time surgical scenes. Our prototype AR navigation system would be highly useful in clinical practice.
Authors: Mina Iskander; Titilola Ogunsola; Rithambara Ramachandran; Richard McGowan; Lama A Al-Aswad Journal: Asia Pac J Ophthalmol (Phila) Date: 2021 May-Jun 01