Use of predictive models to identify patients who are likely to benefit from refraction at a follow-up visit after cataract surgery.

PURPOSE: To develop predictive models to identify cataract surgery patients who are more likely to benefit from refraction at a four-week postoperative exam.
METHODS: In this retrospective study, we used data of all 86,776 cataract surgeries performed in 2015 at a large tertiary-care eye hospital in India. The outcome variable was a binary indicator of whether the difference between corrected distance visual acuity and uncorrected visual acuity at the four-week postoperative exam was at least two lines on the Snellen chart. We examined the following statistical models: logistic regression, decision tree, pruned decision tree, random forest, weighted k-nearest neighbor, and a neural network. Predictor variables included in each model were patient sex and age, source eye (left or right), preoperative visual acuity, first-day postoperative visual acuity, intraoperative and immediate postoperative complications, and combined surgeries. We compared the predictive performance of models and assessed their clinical impact in test samples.
RESULTS: All models demonstrated predictive accuracy better than chance based on area under the receiver operating characteristic curve. In a targeting exercise with a fixed intervention budget, we found that gains from predictive models in identifying patients who would benefit from refraction ranged from 7.8% (increase from 1500 to 1617 patients) to 74% (increase from 250 to 435 patients).
CONCLUSION: The use of predictive statistical models to identify patients who are likely to benefit from refraction at follow-up can improve the economic efficiency of interventions. Simpler models like logistic regression perform almost as well as more complex machine-learning models, but are easier to implement.

Several studies have demonstrated the importance of an intermediate-term follow-up examination after cataract surgery.[12] Patient benefits of a follow-up visit include appropriate diagnosis and management of postoperative complications, such as cystoid macular edema, rebound of postsurgical inflammation, steroid response, and worsening of diabetic retinopathy. Further, more accurate refraction is possible at follow-up because changes in vision due to corneal edema and astigmatic shift typically occur in the weeks following surgery. Globally and especially in developing countries, uncorrected refractive error is a significant source of suboptimal postoperative vision.[345] Importantly, the most commonly used surgical procedure for cataract surgery in developing countries—manual small incision cataract surgery (SICS)—induces a greater need for post-surgical refractive correction than phacoemulsification (PE).[1] The follow-up visit is also an opportunity for the service provider to measure postoperative surgical and visual outcomes, which is useful feedback for continually refining the surgical protocol.[5678]

Postoperative follow-up rates vary greatly across developing countries. In a recent study of 40 centers in ten developing countries in Asia, Africa, and Latin America in which patients were instructed to return for follow-up 40 days after surgery, unprompted follow-up rates ranged from 27% to 93%, with an average of 51%.[8] A study of post-cataract surgery visual outcomes at rural secondary care centers in India reports that nearly 10% of patients did not return for 1–3 week follow-up, and a third did not return for the 4–11 week follow-up.[9] Three other studies in developing countries report follow-up rates of 67.2% at 2 months,[10] 49% at 12 weeks,[11] and 91% at 4–8 weeks.[12]

An important question for improving patient care is how to boost follow-up rates after cataract surgery. In the context of low- and middle-income countries, Ologunde and Rafai[13] advocate that the healthcare provider should attempt to facilitate travel. Studies of noncompliance with follow-up in rural China have found that modest compensation, advertisements, and telephone contact can increase medium-term follow-up rates.[14]

In resource-poor settings, it might be prudent to focus efforts to promote follow-up on those patients who are predicted to be more likely to benefit from the follow-up visit. In this study, we evaluate the usefulness of predictive statistical models in identifying such patients. The aims of our study are: i) to build statistical models to predict which patients are more likely to benefit from refraction at a follow-up visit four weeks after cataract surgery; ii) to compare the predictive performance of alternative statistical model specifications; and iii) to assess the likely clinical impact of predictive models in targeted patient interventions to encourage follow-up.

To develop the predictive models, we used data of all eyes that underwent cataract surgery in 2015 at a large tertiary-care eye hospital in India which performs close to 40% of all cataract surgeries done in the state. Further, the inclusive service design of the hospital wherein patients are sought out in the community and treated irrespective of ability to pay implies that the patient population we study is quite representative of the broader population.

Methods

Ethical approval

The study protocol was approved by the hospital’s Institutional Review Board and adhered to the tenets of the Declaration of Helsinki. Patient confidentiality was maintained by adherence to privacy protocols.

Study design and participants

We conducted a hospital-based retrospective cohort study by analyzing anonymized data of all phacoemulsification (PE) or manual small incision cataract surgeries (SICS) performed at the hospital during 2015, in which patients were asked to routinely follow-up after four weeks at a hospital-affiliated facility. Of the 85,977 surgeries, in 73,728 cases (85.8%), patients returned for a follow-up visit and their data are included in this study. Fig. 1 summarizes the cohort creation and subsequent steps in this study.

Figure 1

Schematic outline of cohort creation and data analysis methodology

Cataract surgery at the hospital

All patients with visually significant cataract preoperatively undergo baseline Snellen visual acuity (VA) assessment, refraction, slit-lamp examination, duct patency, biometry, and systemic comorbidity evaluation. Patients who are advised surgery undergo structured counseling by trained counselors. Surgical procedures at the hospital are performed by its consultants, residents, and fellows. Patients choose one of three types of facilities based on affordability to receive cataract surgical services: paying, subsidized, and outreach camp (free). While paying patients can choose either PE or SICS, almost all the subsidized and camp patients receive SICS.

On the day following surgery, the usual practice is to check Snellen VA with pinhole and conduct a slit-lamp examination. Patients are typically discharged 1 day after surgery. If the operated eye has significant complications, then the patient is discharged 2–3 days later. During discharge, patients are advised to follow-up for a postoperative eye exam on a specific date one month (±5 days) following surgery. All patients are advised to come to the hospital immediately if they experience any sudden loss of vision, redness, or pain. During discharge, the importance of adherence to medications and to follow-up is explained in detail by patient counselors. Each patient is given a written discharge summary relating to follow-up care and medication and the follow-up date. Patients are not charged for follow-up visits at any of the hospital-affiliated sites. In addition, patients included in this study were not provided any incentives or reminders to follow-up. All patients undergo VA testing, subjective refraction, and detailed dilated eye examination using slit-lamp during the follow-up visit.

Data collection

Preoperative, intraoperative, immediate postoperative, and follow-up details of all patients undergoing cataract surgery were obtained from the hospital’s cataract quality monitoring database. These data are captured as an integral part of patient flow and periodically checked for quality. In this study, patients were considered to have complied with the advice to follow-up if they returned for a postoperative examination between 15 and 49 days (both days included) following surgery.

Outcome measure and predictor variables

The key outcome measure is a binary indicator of whether the patient will benefit from refraction at the follow-up visit, which we operationalize as a difference of two lines or more between uncorrected visual acuity (UCVA) and corrected distance visual acuity (CDVA) on the Snellen VA chart.[15] As predictors, we included all variables which are believed to have an association with VA after cataract surgery, and for which data were available in the database. Our approach to identifying and including potential predictors is consistent with our goal of developing predictive models, wherein a causal link between the predictor and outcome is not required.

The predictor variables in each model include for each surgery, patient sex, patient age in years, source eye (left or right), preoperative VA in logMAR units, first-day postoperative VA in logMAR units, and binary indicators of intraoperative complications, first-day postoperative complications, and combined surgeries. Intraoperative and postoperative complications of OCTET grades 2 or 3 (greater severity) are included.[16] A combined surgery is defined as cataract surgery performed along with a cornea-, glaucoma-, or retina-related specialty procedure.

Predictive models and statistical analyses

We considered six alternative statistical models and three “combination” models for this prediction task; these are described in Supplementary Table S1 as M1-M9. The models include both traditional multivariable models like logistic regression, and machine learning models like decision tree and pruned decision tree,[17] random forest,[18] weighted k-nearest neighbor (knn), and a neural network (multilayer perceptron).

Supplementary Table S1

Brief description of predictive statistical models examined in this study

Model	Model Name	Description
M1	Logistic Regression	Description not needed
M2	Decision Tree	Algorithm asks a sequenced set of questions that are designed to maximally separate patients who will benefit from refraction into buckets of no less than 7 patients. Most important questions are asked first and every patient in the same bucket has the same probability of benefiting from refraction.
M3	Pruned Decision Tree	The same as Decision Tree but with fewer questions and more patients in a bucket. Questions are removed if they do not decrease the overall lack of fit by a factor of 0.1%. Results in only four predictor variables being retained in our study.
M4	Random Forest	A combination of 500 decision trees where only three variables can randomly be considered when asking any question in any tree. Patients’ probability of benefiting from refraction is computed by averaging across the 500 decision trees.
M5	Weighted knn	Predicts the probability that a given patient will benefit from refraction as the percentage of the number of 101 patients most similar to this patient who each benefited. Defines similarity by using Euclidean distance between patients, and weights similar patients by a kernel function.
M6	Neural Network	A multi-layered perceptron with three main layers (2 relu activation function with 20 units each, and 1 softmax activation function with 2 units) that uses sparse categorical cross-entropy as a loss function. The black box outputs probabilities that patients will benefit from refraction.
M7	Combination Min	Patient receives a probability of benefiting from refraction equal to the minimum of the six probabilities of benefiting from models M1-M6.
M8	Combination Mean	Patient receives a probability of benefiting from refraction equal to the mean of the six probabilities of benefiting from the models M1-M6.
M9	Combination Max	Patient receives a probability of benefiting from refraction equal to the maximum of the six probabilities of benefiting from the models M1-M6.

The performance of predictive models should be assessed not in the sample in which the model was estimated (“training sample”) but in a separate sample of data after the model is estimated (“test sample”). This is because complex machine learning models are more susceptible to overfitting, which can lead to an overly optimistic estimate of model performance.[19] Consequently, we divided our sample data for each type of surgery (PE and SICS) into two random subsets: two-third in the training sample and one-third in the test sample. Each model was estimated on the training sample data and its predictive performance was measured in both the training sample and the test sample. Analyses were performed with the R programming language with associated statistical packages. The neural network was implemented using the Keras package in R with Tensorflow as the back-end for computational efficiency.

We measure the validity of models as their ability to discriminate between surgeries in which patients benefit from refraction versus those in which patients do not benefit, using the test sample only. We use receiver operating characteristic (ROC) curve analysis. ROC curves plot true positive rates against false-positive rates for patients predicted to benefit from refraction. One ROC curve exists for each model with each point on the curve representing a clinical decision. A model is dominant compared to others if its true positive rate is highest for any given false positive rate. The area under the ROC curve (AUC) is the probability of a model ranking a randomly chosen patient who benefits from refraction higher than a randomly chosen patient who does not benefit from refraction. Thus, high AUCs are associated with better model performance in discrimination.

Clinical impact of predictive models

We define “gainers” as patients who will benefit from refraction at the follow-up visit. The primary goal of the predictive models is to identify gainers. To assess the clinical performance of alternative predictive models in terms of this task, we construct the following exercise. Say a fixed total budget of $B is available to intervene to encourage patients to follow-up, and that an intervention costs $c per patient. For instance, Meltzer et al. (2015)[15] report that the estimated mean cost to a patient in India of a spontaneous follow-up visit, including round-trip transportation cost, food and living expenses, and loss of wages for the patient and accompanying persons, was $8.34. This implies that M = $B/$c patients can be targeted with the intervention. The question of interest is who these M patients should be. Ideally, the intervention budget should be spent entirely on gainers.

Say the test sample includes NTest patients, of which benefited from refraction, i.e., are gainers. This implies that if we choose M ≤ NTest patients from the test sample at random, we would expect a fraction to be gainers in the chosen set. For each estimated predictive model, we sort patients in the test sample in decreasing order of the predicted probability of being a gainer, and then compute the fraction k based on the number of gainers included in the top M patients. We do this exercise for different values of M ranging from 1 to NTest. A “Gains Chart” displays the fraction k on the vertical axis, and the fraction M/NTest on the horizontal axis, for each predictive model (a gains chart is closely related to the ROC curve, but carries extra information.[20]) For any point on the horizontal axis, a larger k indicates that a larger number of gainers are targeted with the intervention; hence, this implies better performance from the perspective of clinical impact. Now models can be compared visually in terms of their ability to maximize the number of patients who would benefit from refraction, in any targeted set of M patients.

Results

Sample description

Table 1 shows descriptive statistics of the patient sample. As expected, we find that the profiles of patients in the training sample are very similar to those in the test sample, for both PE and SICS. However, there are important differences in the patient profiles of PE versus SICS. Notably, while 21% of 16,412 PE patients in the training sample benefit from refraction at follow-up, the proportion is much higher at 71.5% of 30,787 patients for SICS.

Table 1

Descriptive statistics of patient sample

Variables	Phacoemulsification (n=24,617)				SICS (n=46,180)
	Training (n=16,412)		Test (n=8205)		Training (n=30,787)		Test (n=15,393)
Benefitted from Refraction
Yes	3448	21.0%	1711	20.9%	22,000	71.5%	11,034	71.7%
No	12,964	79.0%	6494	79.1%	8787	28.5%	4359	28.3%
Patient Type
Camp and Free	427	2.6%	203	2.5%	27,448	89.2%	13,657	88.7%
Pay	15,985	97.4%	8002	97.5%	3339	10.8%	1736	11.3%
Patient Age (years)
Median	61		61		60		60
Interquartile Range	54.0-67.0		54.0-67.0		55.0-66.0		55.0-67.0
Patient Sex
Female	8211	50.0%	3989	48.6%	17,739	57.6%	8810	57.2%
Male	8201	50.0%	4216	51.4%	13,048	42.4%	6583	42.8%
Operated Eye
Left	8009	48.8%	3973	48.4%	14,916	48.4%	7449	48.4%
Right	8403	51.2%	4232	51.6%	15,871	51.6%	7944	51.6%
Preop UCVA (logMAR)
Median	0.778		0.778		1.079		1.079
Interquartile Range	0.602-1.079		0.602-1.079		1.000-1.778		1.000-1.778
Discharge Pinhole (logMAR)
Median	0		0		0.18		0.18
Interquartile Range	0-0.18		0-0.18		0-0.30		0-0.30
Intraop Complications
Yes	205	1.2%	87	1.1%	342	1.1%	198	1.3%
No	16,207	98.8%	8118	98.9%	30,445	98.9%	15,195	98.7%
Day-1 Postop Complications
Yes	401	2.4%	218	2.7%	525	1.7%	292	1.9%
No	16,011	97.6%	7987	97.3%	30,262	98.3%	15,101	98.1%
Combined Surgery
Yes	182	1.1%	96	1.2%	28	0.1%	7	0.0%
No	16,230	98.9%	8109	98.8%	30,759	99.9%	15,386	100%

SICS – small incision cataract surgery; UCVA – uncorrected visual acuity; logMAR – log of the minimum angle of resolution; Intraop – intraoperative; postop – postoperative

Model validation

In Fig. 2, we show the area under the ROC curve (AUC) and its 95% confidence interval for each model in the training sample and the test sample, for PE and SICS separately. We note that due to overfitting, two of the six models provide overly optimistic estimates of their performance in the training sample relative to their performance in the test sample; this is indicated by non-overlapping 95% confidence intervals of the training and test samples. These two are the more complex machine learning models: random forest and knn, in both PE and SICS. We note that the neural network was specified with a low number of layers (3) and neurons in each layer (20, 20, and 2, respectively) and did not overfit in the test sample. Notably, the performance of all six models in the test sample does not vary much: for both PE and SICS the AUC of the six models ranges from 0.61 to 0.65.

Figure 2

Area under the curve (AUC) for receiver operating characteristic curve for phacoemulsification (upper panel) and small incision cataract surgery (lower panel). Error bars indicate 95% confidence intervals

Clinical impact

In Fig. 3, we show the Gains Charts resulting from the estimated predicted models in the training and test samples for PE and SICS, respectively. For ease of interpretation, we show only three curves. The curve marked “Random” is the 45° line and serves as a benchmark. The curves marked “Frontier” and “Minimum,” respectively, show the performance of the best- and worst-performing models for each point on the horizontal axis. Consistent with the AUC results discussed previously, the Frontier and Minimum curves are very similar in the test sample (though not in the training sample) for both PE and SICS, indicating that all models exhibit similar performance in terms of impact for each possible clinical decision.

Figure 3

Gains chart for Frontier model (best-performing for each point on X-axis) and Minimum model (worst-performing for each point on X-axis) for phacoemulsification in training sample (upper left panel) and test sample (lower left panel) and for small incision cataract surgery in training sample (upper right panel) and test sample (lower right panel)

Next, we compare the Frontier curve to the benchmark Random curve when (say) 50% of the patients in the test sample are targeted; this is indicated by the dashed vertical line. As expected, the Random curve shows that if the targeted patients were selected at random, 50% of patients who would benefit from refraction would be included in the targeted set. By contrast, if the Frontier model were used, close to 67% [PE, left lower panel in Fig. 3] and 56% [SICS, right lower panel in Fig. 3] of patients who would benefit from refraction would be included in the targeted set. Thus, the predictive models provide large benefits for PE and modest benefits for SICS.

To further demonstrate the clinical benefits of predictive models, we conduct an exercise wherein we assume values of an intervention budget ranging from $10,000—$60,000 [results in Table 2]. For each level of the budget, given an assumed cost per intervention of $8.34 based on Meltzer et al.,[15] we determine the number of patients who can receive the intervention. We then compute for PE and SICS separately, the number of gainers who would be captured in the set of intervened patients, if they were chosen randomly from the test sample, versus chosen with the best predictive model. We see that the use of the predictive model increases the number of gainers captured. For PE, this increase ranges from 7.8% (1500 to 1617 patients) to 74% (250 to 435 patients), while for SICS the gains are smaller and range from 12.4% (5156 to 5797 patients) to 18.2% (859 to 1015 patients).

Table 2

Clinical benefits from use of predictive models in targeting patients to reach “gainers” in the test samples

Budget	Cost per intervention	Number of patients targeted	Phacoemulsification (Number of Gainers in Test Sample=1711 out of 8205)			SICS (Number of Gainers in Test Sample=11,034 out of 15,393)
			# Gainers reached: Random	# Gainers reached: Frontier Model	% Improve-ment (PE)	# Gainers reached: Random	# Gainers reached: Frontier Model	% Improve-ment (SICS)
$10,000	$8.34	1199	250	435	74.0	859	1015	18.2
$20,000	$8.34	2398	500	776	55.2	1718	1996	16.2
$30,000	$8.34	3597	750	1049	39.9	2578	2972	15.3
$40,000	$8.34	4796	1000	1277	27.7	3437	3928	14.3
$50,000	$8.34	5995	1250	1466	17.3	4297	4854	13.0
$60,000	$8.34	7194	1500	1617	7.8	5156	5797	12.4

PE – phacoemulsification; SICS – small incision cataract surgery

For both SICS and PE, in the models that did not overfit, the following four predictor variables were found to be important for predictions in our models, with varying relative importance across models: patient age, patient sex, patient type, discharge vision.

Discussion

Uncorrected refractive error after cataract surgery is a major concern especially in developing countries. One important source of this problem is low patient follow-up rates. In resource-constrained settings, it is important to decide which patients to prioritize to encourage follow-up. In this study, we explored the role of predictive models in identifying those patients who are likely to benefit from refraction during the follow-up visit after cataract surgery.

Our most important finding is that predictive models can be effective in identifying such patients. Say budgets allow one in two patients to be targeted with an intervention such as a reminder phone call or a transport subsidy to encourage follow-up. In our data, we found that, relative to not using a predictive model, the use of predictive models increased the percentage in the targeted group of those who would benefit from refraction by 34% (i.e., go up from 50% to 67%). The analogous benefit in the case of SICS patients is 12% (increase from 50% to 56%).

As discussed, there are large differences in the performance of predictive models between PE and SICS patients. While refraction will only benefit about 2 in ten patients who undergo PE, predictive models can substantially increase the efficiency of identifying these patients for targeted intervention. By contrast, as many as 7 in ten SICS patients can benefit from refraction; however, the gains from predictive models for SICS were modest. This suggests the need for further work in identifying explanatory variables that would improve the predictive ability of models for SICS, which is often the preferred technique in developing countries.[21]

While our findings establish the feasibility of a robust predictive model, more work needs to be done to develop a tool for patient-level predictions that can be integrated into a clinical caregiving process. Such a tool could be used to identify at surgical discharge specific patients who are predicted to gain due to refraction. Identifying such patients would also help in setting patient expectations about the need for corrective spectacles.

In this study, we used predictor variables for which data were routinely available in the surgical quality monitoring database. We recommend that hospitals that implement our proposed predictive models collect data on at least the four variables that were found to be important in the prediction task. With our data, we found that machine learning models like random forests, weighted k-nearest neighbor, and neural networks did not outperform logistic regression. This suggests the limited role of significant nonlinear relationships between predictors and the outcome of interest. The fact that traditional models like logistic regression provided comparable predictive performance in our setting is beneficial from a pragmatic point of view. Traditional models require less data and are simpler to calibrate and implement.

As opposed to summary statistics, machine learning models find patterns between multiple factors and an outcome, simultaneously. For example, a machine learning model may learn that female outreach patients aged 50+ years with a preoperative UCVA <=6/12 are more likely to benefit from refraction at follow-up. However, the same might not be true for patients 50+ years old regardless of other patient characteristics. Summary statistics cannot efficiently capture patterns between multiple factors at once and project them into the future. However, using machine learning models like neural networks to identify and interpret the important factors that predict the gainers is computationally complex, and depends in part on the skill of the analyst. We also found that some machine learning methods like random forest and weighted k-nearest neighbor were susceptible to substantial overfitting in the training sample, reinforcing the importance of assessing the validity of predictive models in test samples.

Of course, artificial intelligence and deep learning models have an important role to play in ophthalmology in terms of detection of eye diseases (diabetic retinopathy, glaucoma, age-related macular degeneration, etc.), predicting progression (myopia, keratoconus, and glaucoma), and evaluating treatment outcomes (anti-vascular endothelial growth factor).[222324] Our use of these models for equitable distribution of resources to achieve socioeconomic benefits is novel.

Conclusion

The use of predictive statistical models to identify patients who are likely to benefit from refraction at follow-up can improve the economic efficiency of interventions to encourage follow-up. Simpler models like logistic regression perform almost as well as more complex machine-learning models, but are easier to implement. With currently available data the benefits of predictive models were found to be larger for patients undergoing phacoemulsification than SICS.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.