Real-time artificial intelligence evaluation of cataract surgery: A preliminary study on demonstration experiment.

PURPOSE: We demonstrated real-time evaluation technology for cataract surgery using artificial intelligence (AI) to residents and supervising doctors (doctors), and performed a comparison between the two groups in terms of risk indicators and duration for two of the important processes of surgery, continuous curvilinear capsulorhexis (CCC) and phacoemulsification (Phaco).
MATERIALS AND METHODS: Each of three residents with operative experience of fewer than 100 cases, and three supervising doctors with operative experience of 1000 or more cases, performed cataract surgeries on three cases, respectably, a total of 18 cases. The mean values of the risk indicators in the CCC and Phaco processes measured in real-time during the surgery were statistically compared between the residents' group and the doctors' group.
RESULTS: The mean values (standard deviation) of the risk indicator (the safest, 0 to most risky, 1) for CCC were 0.556 (0.384) in the residents and 0.433 (0.421) in the doctors, those for Phaco were 0.511 (0.423) in the residents and 0.377 (0.406) in the doctors. The doctors' risk indicators were significantly better in both processes (P = 0.0003, P < 0.0001 by Wilcoxon test).
CONCLUSION: We successfully implemented a real-time surgical technique evaluation system for cataract surgery and collected data. The risk indicators were significantly better in the doctors than in the resident's group, suggesting that AI can objectively serve as a new indicator to intraoperatively identify surgical risks. Copyright:

Introduction

Cataract surgery is one of the most frequently performed surgical procedures in the world,[1] and it is a medical procedure of primary importance in the field of ophthalmology. This highly developed procedure has been supported by many factors,[2] including advances in cataract surgery techniques,[3] advances in ultrasonic emulsification and aspiration devices, advances in intraocular lenses,[4] the development of the lens power calculation formula,[5] and the development of viscoelastic substances and intraocular lens reflux fluid. Cataract surgery has become common in developed countries, and some have even pointed out the harmful effects of undergoing the procedure too early.[6] On the other hand, cataracts continue to be the leading cause of blindness in the world.[1] One of the contributing factors is existing public health problems in developing countries, causing congenital cataracts, which lead to blindness. Another major factor that has been pointed out is the overwhelming shortage of ophthalmologists who can perform cataract surgery.[7] Although many volunteer ophthalmologists in developed countries visit areas, such as Africa, where there is a shortage of ophthalmologists to perform surgeries, it is merely a drop in the bucket; the situation has not improved over a lengthy period.[8] The importance of training ophthalmologists and even surgeons who can perform cataract surgery has driven the development of computer-based surgical simulators for training purposes.[910]

Surgical training presents many challenges not only in developing countries but also in developed countries.[11] This is partly because surgical training environments for residents have become more severe with the progress of cataract technology. Over the years, the duration of standard surgery and the postoperative recovery period has been shortened.[12] It is impossible for residents with very few surgical experiences to successfully perform surgeries under such conditions. In countries where the informed consent process is strictly performed, one of the greatest concerns of patients is the number of surgical procedures the surgeon has performed in the past. In developed countries where patients have a high awareness of their rights, such as the United Kingdom, it is legally required to disclose the number of procedures that a doctor has performed.[13] One report[11] shows that only 56.3% of patients agreed with the residents’ participation in surgery after undergoing the clearly-communicated informed consent process. As clinical practice in cataract surgery continues to develop in many aspects, advances in training for cataract surgery have continued to be a major concern. It has been pointed out that, not only in cataract surgery[14] but also in gastric bypass surgery,[15] the number of surgical procedures previously performed is actually one of the indicators to reduce risk rates and the risk of complications. How to make surgeries performed by less experienced surgeons safer is an ever-present and important issue in the medical community.

A surge in the social application of artificial intelligence (AI) technology has been fueled since a research team won, by a substantial margin, a visual recognition challenge in 2012 with a deep learning model, leaving all other teams behind by more than 10%.[16] AI applications to the field of ophthalmology began after a Google team reported a diabetic retinopathy diagnosing AI model in 2016.[17] Since then, there have been many AI diagnostic models for major fundus diseases, including glaucoma,[18] age-related macular degeneration,[19] and retinal detachment.[20] Meanwhile, IDx released its diabetic retinopathy diagnosis AI (2018), which was the first AI diagnostic device approved by the Food and Drug Administration.[21] This release indicated that the field of ophthalmology was leading in AI diagnosis. The primary characteristics of deep learning are that it analyzes unstructured data, and it has been used for feature analysis even in video media which consists of a stack of still images. For YouTube, AI that automatically eliminates inappropriate videos from a large number of uploaded videos is an indispensable core technology.[22] The application of AI to surgical videos has been seen for a relatively long time, mainly in cataract surgery. Representative studies include the real-time extraction of important surgical phases in cataract surgery,[23] and a study based on a former study of real-time risk indicator display systems,[24] both by Morita et al. They reported that their model could predict from example surgical videos the incidence of subsequent intraoperative complications with a response rate of 92%.

Many previous studies[252627282930] on variety types of videos analysis using AI employed recorded entire videos to perform postanalysis. In the present study, we ran a real-time cataract surgery AI risk rating system for cataract surgery performed by ophthalmologists and residents, and statistically examined the results.

Materials and Methods

Overview

The single cataract surgeries performed in October 2021 at the Department of Ophthalmology, Saneikai Tsukazaki Hospital, were included in our analysis. A total of 18 cataract surgeries were performed by each of three residents with experience of <100 cataract surgeries and three ophthalmologists (“doctors”) with experience of more than 1000 cataract surgeries. During the surgery, a real-time cataract risk indicator computing AI model was run, and risk indicators were recorded. The risk indicators were calculated by taking five moving averages (sampling was tenth per second) of the estimated value obtained by the important process recognition model, which is multiplied by the estimated value obtained by the surgical risk recognition model for each process. The results were also recorded as log data in a comma-separated values file. In addition, the screen recording function on a personal computer was used to record the performance of the real-time cataract risk rating AI model by having the following three windows displayed on the computer screen: a microscopic field video, a log for values calculated by AI, and a real-time graph displaying risk rates. For the two processes of the cataract surgery, continuous curvilinear capsulorhexis (CCC) and phacoemulsification (Phaco), the start and end times of each process were extracted from the screen recording video by H. T. The risk indicators were calculated for the two processes, and statistical analysis was performed to find a difference between the residents’ group and the doctors’ group. The statistical analyses were performed by JMP Ver 14.0.2. The study was conducted in accordance with the Declaration of Helsinki and was approved by the Ophthalmology Ethics Committee of Tsukazaki Hospital. (Approval Number, 1806) Written consent forms were obtained from patients.

Artificial intelligence model

Details on the AI models are described in the previous report;[24] below is a summary. The algorithm used in this study was constructed by a two-stage model. The first-stage model, Model A, recognizes a total of two phases of the cataract surgery, one is from the start of CCC to the end of Phaco, which are referred to here as important processes, and the other is the remainder of the phase. A total of 422,559 still images extracted from cataract surgery videos of 425 cases recorded at a resolution of 1920 × 1080 pixels and 30 frame per second (FPS) were used to construct Model A. This Model A provided the value of 1, if it determined that the image was the closest to the phase that is from the start of CCC to the end of Phaco; the model provided the value of 0, if it determined that the image was most likely to be the rest of the phase. The recognition performance of Model A was 91.3%. Next, we built the second-stage model, Model B, which identifies the degree of risk as to whether or not problematic events occur during the phase from the beginning of CCC to the end of Phaco. The purpose of this model is to calculate the degree of risk and present it to the surgeon during the course of surgery; therefore, the calculation must be performed fast. To achieve the required speed, the training data for Model B were downsampled to a resolution of 299 × 168, and the extraction speed was also reduced to 10 FPS. A total of 156,170 images were used to create the risk calculation model. The following nine problematic events were used to train Model B: vitreous prolapse, capsule rupture, damage to the iris, iris prolapse, rupture of the Zonule of the Zinn, dropped nucleus, discontinuous CCC, CCC tear, and wound suture. Model B provided the value of 1, if it determined that the image was closest to a particular problematic event, and it provided the value of 0, if it determined that it was farthest. These two models were run at the same time, and the risk indicators were calculated in real-time by multiplying the indicator value of Model A by that of Model B. At the time of real-time analysis, Model A was also run at 10 FPS. The integrated risk indicators (IRI) were calculated by real-time analysis in the following way. For example, when Model A determined that the surgical phase was the phase between the start of CCC to the end of Phaco by providing the indicator value of 0.9991, and Model B determined the degree of risk to be 0.8811, then IRI would be 0.9991 × 0.8811 = 0.8803. Using the threshold value of 0.989 (area under the curve, 0.970), which represents a maximum identification ability in IRI, the model was able to predict events that lead to problems with a probability of 92%, and 42 out of 44 problematic events in advance. Inception V3[31] was used to configure the neural network for Models A and B. This model was trained by initializing each parameter with trained parameters in the ILSVRC 2012 dataset.[32] The training parameters were set to a batch size of 32, the loss function of categorical cross-entropy, the optimization function of momentum stochastic gradient descent (SGD) (learning rate, 0.0001; momentum, 0.9), and the number of epochs of a maximum of 300. The network was trained on a system with two NVIDIA GTX 1080 Ti GPUs, and the evaluation was performed on a single GPU.

Implementation setting

All cataract surgeries were performed with an ophthalmic surgical microscope (OPMI Lumera 700, Zeiss, Oberkochen, Germany), and high-definition video taken with the built-in 3CCD camera was output via an serial digital interface (SDI) cable. It was converted to an HDMI signal by converter 1 (Mini Converter-SDI to HDMI 6G, Blackmagicdesign, Melbourne, Australia). The HDMI signal was converted to USB by a video converter 2 (AVT-C878, AVerMedia Technologies Inc, New Taipei City, Taiwan) and input to a GPU for analysis (ELSA GeForce RTX 2070 S. A. C, ELSA Japan, Tokyo, Japan). For analysis video recording, the monitor image output from the GPU machine as a digital visual interface signal was converted to an HDMI signal via a conversion cable, and then output via USB with the above-mentioned video converter 2 for screen capture laptop computer (YOGA 720, Lenovo, Morrisville, USA) and Snipping Tool (Windows10, Microsoft, Albuquerque, USA) was used to record screen [Figure 1].

Figure 1

Setting of real-time artificial intelligence evaluation of cataract surgery. (a) A full view of the setting; the part with the white arrow is the artificial intelligence system. (b) The entire artificial intelligence system; (a) Analysis GPU machine, (b) Analysis monitor, (c) Video signal converter, the part with the white arrow is a recording monitor. (c) Artificial intelligence system monitor display screen; (d) Risk indicator log, (e) Microscopy field image, (f) Risk indicator graph

Severity of cataracts and continuous curvilinear capsulorhexis and phacoemulsification procedures

All cataracts in the study were either less than the moderate grade of nuclear cataract listed in the World Health Organization[33] standards, or cortical cataracts. Mydriasis was good in all cases (pupil diameter of 8 mm or more at the start of surgery). The incision was made at the 9 o’clock or 10 o’clock position, and the two forceps used for CCC were the Ikeda[34] and Inatomi[35] types. Phaco techniques used were the prechopper,[36] the phacochopper,[37] and the divide-and-conquer[38] techniques. All of these techniques were included in the training data when the model used in this study was created. The surgical details are shown in Table 1. The same Phaco device (CENTURION, Alcon, Fort Worth, USA) was used for all cases.

Table 1

Surgical details

	Wound	CCC forceps	Phaco	Grades of cataract
Dr.T	9 o’clock	Inatomi	Prechopper	G2, cortical×2
Dr.S	10 o’clock	Ikeda	Dvidid Concour	G2×3
Dr.N	9 o’clock	Inatomi	Phacochopper	G3×2, cortial
Resident A	10 o’clock	Ikeda	Dvidid Concour	G2×3
Resident B	10 o’clock	Ikeda	Dvidid Concour	G2×2, cortical
Resident C	10 o’clock	Ikeda	Dvidid Concour	Cortical×2, G1

WHO=World Health Organization, Gn=Grade n of nuclear cataract of WHO standards, CCC=Continuous curvilinear capsulorhexis

Results

No intraoperative complications were observed in all cases. The patient demographic data are presented in Table 2. Figure 2 shows the distribution of risk indicators in the entire CCC and the entire Phaco processes in all cases. Particularly, the results show that the risk indicators fluctuated around a value close to 1 over the course of surgery in residents. Statistical analysis of these results is as follows.

Table 2

Patient demographics

	n	Age (SD) years	Female, n (%)	Axial length (SD) mm
Doctors	9	75.2 (4.3)	5 (56)	25.0 (0.99)
Residents	9	74.3 (5.3)	4 (44)	24.7 (1.04)
All	18	74.8 (4.7)	9 (50)	24.9 (0.99)

SD=Standard deviation

Figure 2

Changes of risk indicators for two processes in all cases

For CCC, the mean value (standard deviation, [SD]) of the residents’ risk indicator was 0.556 (0.384) with a median of 0.637, and their mean duration (SD) was 174 (261) seconds with a median of 63 s. On the other hand, the mean value (SD) of the doctors’ risk indicator for CCC was 0.433 (0.421) with a median of 0.285. The doctors’ mean duration (SD) was 32.3 (12.1) seconds with a median of 22.5 s. Significant differences were observed between the two groups in the risk indicators and duration, both of which were better with the instructors (P = 0.0003, P = 0.0015 by Wilcoxon test).

For Phaco, the mean value (SD) of the residents’ risk indicator was 0.511 (0.423) with a median of 0.649, and their mean duration (SD) was 517 (339.6) seconds with a median of 471 s. On the other hand, the mean value (SD) of the doctors’ risk indicator for Phaco was 0.377 (0.406) with a median of 0.177. The doctors’ mean duration (SD) was 107 (63.6) seconds with a median of 79 s. Again, significant differences were observed between the two groups in the risk indicators and duration, both of which were better with the doctors (P < 0.0001, P = 0.0041 by Wilcoxon test). Figure 3 shows the mean risk indicators, and Figure 4 shows the mean duration of the procedure.

Figure 3

Comparison of mean risk indicators for continuous curvilinear capsulorhexis and phacoemulsification processes between the two groups

Figure 4

Comparison of mean duration (second) for two processes between the two groups

In the Figure 2, the left column shows the doctors’ data (First to third from the top is Dr. T, fourth to sixth is Dr. S, and seventh to ninth is Dr. N.), and the right column shows the residents’ data (First to third from the top is Resident A, fourth to sixth is Resident B, and seventh to ninth is Resident C). The graphs show the changes in the risk indicators for CCC and Phaco processes over the course of the procedure in 18 cases. The vertical axis shows the risk rates, and the horizontal axis shows the passage of time from the start of surgery. As for the risk indicator, the minimum value is 0, and the maximum value is 1, which represents the highest risk. The horizontal axis is the elapsed time from the start of the surgery (minutes: Seconds: milliseconds).

Statistically significant differences were observed in the mean risk indicators (value of 0–1, where 1 represents the highest risk) for both CCC and Phaco (nuclear extraction phase) processes between the two groups, residents versus doctors who have operative experience of more than 1000 cases (Wilcoxon test).

Statistically significant differences were observed in the mean duration (second) for both CCC and Phaco (nuclear extraction phase) processes between the two groups, residents versus doctors who have operative experience of more than 1000 cases (Wilcoxon test).

Discussion

Our results demonstrate that the degree of risk calculated by our real-time cataract surgery risk rating model was significantly better in doctors for the two important processes, CCC and Phaco, of the surgery. The scatter plots showing the risk indicators for the entire course of these two processes show a clear difference between the groups.

Training in cataract surgery has advantages and disadvantages. Since the entire surgical field can be recorded under a microscope, and the operation of a supervising doctor is completed in a very short time, training in cataract surgery is easy as residents can learn from the video in advance. On the other hand, the biggest challenge in cataract surgery training is the fact that the surgery needs to be performed when the patient is awake and experiencing extreme anxiety.[39] It would be inappropriate in actual clinical practice for the supervising doctor to give a verbal warning to the resident that a particular technique is dangerous, or to take turns between the resident and the doctor rather frequently. The system we present in this study makes it possible to visually inform when the risk rate increases. If a monitor is installed in a way so that the supervising doctor can glance at it to check the risk rate graph, the supervising doctor and the resident could make sound judgments nonverbally based on the risk rating data and safely take turns as needed without informing the patient of the situation. Furthermore, we believe that sharing such objective predictive indicators for surgical evaluation in real-time with both the supervising doctor and the resident is valuable in having a mutual agreement more quickly.

In addition, these risk indicators may be useful in objectively understanding the progress of the learning curve when a supervising doctor is providing technical feedback using recorded data after surgery. If the goal is to simply shorten the surgical duration, which is a typical indicator, the risk of intraoperative complications may increase by compromising the accuracy in incision and CCC steps. On the other hand, in order to improve our risk indicator, one is required to perform more carefully without mistakes. This indicator may guide residents to the traditional view that the more accurate that techniques become, the shorter the duration of the surgery will be as a result.

The present study has many limitations. First, it does not outweigh the usefulness of pretraining, such as CG simulation-based training, in which a technique is learned before the actual patient is operated upon. One report[40] shows that the rate of complications during the cataract surgery training period was reduced in a greater amount in the group trained with EyeSi, a leading surgical technique training application, than the group not trained with the application. Our model needs to be examined for its application to pretraining with animal eyes such as pig eyes, or how the use of the model can positively impact the training. Moreover, our model is neither a method to evaluate surgical techniques by giving scores, such as the Objective Structured Assessment of Cataract Surgical Skill,[41] nor does it have a function to provide guidance for desirable techniques. Furthermore, the processes evaluated by our model are limited to CCC and Phaco. Surgical errors can happen in other processes such as irrigation and aspiration and lens insertion, and the creation of incisions and hydration are also very important for cataract surgery. More research is needed to examine how the model can evaluate the techniques throughout the entire cataract surgery. Furthermore, a further study with a large number of cases is needed to rate the risk according to the level of technical difficulty presented by a given surgery, such as nuclear hardness.

In this study, we did not compare the evaluation by AI with the precise technical evaluation by other methods. Therefore, it does not mean that the AI-based technology assessment method represents the actual intraoperative risk precisely. Furthermore, the rating of 1, which is the highest risk, appeared many times during this experiment. There is a possibility that the risk assessment is likely to be too strict than necessary. In addition, there was a lapse of time in which the risk values differed significantly among the experienced participants. Further development of the AI model is needed, such as reviewing the training data and adjusting the threshold settings.

Today, the research and development around AI technology is soaring in all sectors. These technologies enable applications other than surgical field video information. Microscopic surgical field information does not capture the whole picture of cataract surgery. What is recorded in the microscopic surgical field video is not the movement of the surgeon's hand itself, but the movement of the instrument as a result of the movement of the hand, and the movement of the ultrasonic tip and the nucleus, as a result of the amount of depression of the foot pedal by the surgeon. How the surgeon's hands and fingers move or how much the foot pedal is being pressed cannot be understood from the surgical video alone. An AI evaluation system that collects information on finger movements and foot pedals and organically integrates it with information from the microscopic field can dramatically streamline surgical training. The application of AI to cataract surgery evaluation systems has just begun, and we are enthusiastic about further expanding our research in the future.

Conclusion

In this study, we successfully implemented a real-time surgical technique evaluation system for cataract surgery and collected data. The analysis of risk indicators for the surgical techniques of the CCC and Phaco processes showed that the mean values of each of the entire processes for the supervising doctors were significantly lower, meaning that their techniques were safer than those of the residents. In the future, we would like to expand our research and develop an indicator that captures cataract surgery techniques from multiple perspectives, such as the movements of the surgeon's fingertips and hands, as well as foot pedal information, rather than merely surgical field information. We are eager to pursue the possibilities that AI technology can offer for streamlining cataract surgery training.

Financial support and sponsorship

This research was supported by AMED under Grant Number JP21vk0124003 and Japanese Health and Labor Sciences Research Grants Number JP 20AC1006.

Conflicts of interest

The authors declare that there are no conflicts of interests of this paper.