Validation of an Algorithm for the Prediction of Diabetic Retinopathy in Type 1 Diabetic Patients in a Spanish Population.

Aim: The aim of the present study was to build a clinical decision support system (CDSS) that can predict the presence of diabetic retinopathy (DR) in type 1 diabetes (T1DM) patients. Material and Method: We built two versions of our CDSS to predict the presence of any-type DR and sight-threatening DR (STDR) in T1DM patients. The first version was trained using 324 T1DM and 826 T2DM patients. The second was trained with only the 324 T1DM patients.
Results: The first version achieved an accuracy (ACC) = 0.795, specificity (SP) = 83%, and sensitivity (S) = 65.7% to predict the presence of any-DR, and an ACC = 0.918, SP = 87.1% and S = 87.8% for STDR. The second model achieved ACC = 0.799, SP = 87.5% and S = 86.3% when predicting any-DR and ACC = 0.937, SP = 90.9% and S = 83.0% for STDR.
Conclusion: The two models better predict STDR than any-DR in T1DM patients. We will need a larger sample to strengthen our results.

Introduction

Diabetes mellitus (DM) is a global disease whose incidence is increasing rapidly. In Europe alone, 59 million people are affected (2019 data) by Type 1 (T1DM) and Type 2 (T2DM) diabetes mellitus, a number that is predicted to rise by 15% to 68 million by 2045.1 These diseases bring their own complications, such as microangiopathy, of which diabetic retinopathy (DR), an ocular manifestation, is the most frequent cause of poor vision and blindness for these patients. Science associations recommend that ophthalmology units, with the support of general practitioners, endocrinologists, and paediatricians, carry out regular screening of the retina at various frequencies depending on the risk of progression to DR.2,3

Screening for DR needs to be carried out annually,2,3,4 and early detection is crucial for establishing the most effective treatments. For patients with DM but without DR, it is very important to monitor each patient according to their level of risk.5–7

To personalize DR screening programmes, we have developed an algorithm named RETIPROGRAM, a diagnostic aid system for T2DM patients.9 The aim of present study was to build a new algorithm that can include T1DM patients.

Methods

Setting

The reference population in our area is 247,174. The total number of DM patients registered with our Health Care Area (University Hospital Saint Joan, Tarragona, Spain) is 17,792, 5% of whom are T1DM patients. Our DR screening programme has been ongoing since 1st January 2007, when we started offering annual retinography to our DM patients, including T1DM patients. The screening programme is described more completely elsewhere.10

Design

The aim of the present study was to build an effective Clinical Decision Support System (CDSS) for predicting diabetic retinopathy in T1DM patients, because the one we created originally (RETIPROGRAM) was not built for T1DM patients. To achieve this, we decided to build two models and evaluate which one worked better. The first model included both T1DM and T2DM patients (324 and 826, respectively) as recommended by our epidemiologist, and the second CDDS included only T1DM patients.

Ethics

The study was approved by the Ethics Committee of Institut d’Investigació Sanitària Pere Virgili.

Model Construction

Our CDSS for diabetic retinopathy (RETIPROGRAM) was built from a sample of patients taken from our own DM population of 17,792 under the care of Hospital Universitari Sant Joan de Reus. Briefly, the algorithm was based on a statistical analysis that uses fuzzy decision trees, with a distribution of different risk variables at the nodes of multiple trees. Our algorithm has 200 trees with three variables at each node used for prediction.11 The variables considered significant were: age, sex, body mass index, duration of DM, treatment of DM, arterial hypertension control, HbA1c in %, microalbuminuria, and glomerular filtration rate determined by the CKD-EPI formula (Figure 1 shows the finished model).12 Figure 2 shows how the system works. First, a patient’s variables are introduced to the algorithm, then the system analyses the data and gives the percentage probability (%) that the patient either has retinopathy or not, the result not being indeterminate rather than clearly defined. Finally, in the output, the system indicates the percentage probability risk of DR to the patient: no risk, or low, moderate or high risk. Based on each of these 4 results, or outputs, the system indicates when the patient should go to the ophthalmologist. In the present study we trained the algorithm in two models. In the first, 195 T1DM patients and 495 T2DM patients were randomly chosen from our screened population and, from those, the testing step was carried out using a sample of 129 T1DM and 331 T2DM, again randomly chosen. In the second model, the algorithm was re-trained with only the T1DM patients and 195 T1DM were randomly chosen for training the model and the remaining 129 T1DM patients for testing.

Figure 1

An example of the RETIPROGRAM application for a patient at high risk of developing diabetic retinopathy.

Figure 2

An example of a decision tree. We describe the process in four columns. The first column indicates the input of a patient’s data (we introduce the nine variables that we determined), the second column shows the fuzzy decision tree model, the third column indicates the probability in percentage of having retinopathy or not, and if the algorithm if the result is indeterminate it is shown as a percentage. For example in this case patient has an 88% probability of no risk of retinopathy. The last column shows the recommendation for the next visit accordingly.

Inclusion Criteria

Patients with T1DM or T2DM, as diagnosed and listed at Hospital Universitari Sant Joan de Reus.

Exclusion Criteria

Patients with other types of diabetes.

Diagnosis of Diabetic Retinopathy

Two senior retina ophthalmologists diagnosed the condition based on EURODIAB guidelines,13 using a non-mydriatic fundus camera, which took two 45° field retinal photographs (TOPCON® TRC-NW6S), one centred on the macula and the other on the temporal side of the optic nerve. DR was confirmed by the ophthalmologist after a hospital visit with new retinographs. The diagnosis was confirmed when microaneurysms were present in the retinographs in the absence of other known causes of the changes, and the DR severity was classified according to the International Diabetic Retinopathy Classification.14 For the statistical study we classified DR as follows: any-DR, which included all forms of DR (mild, moderate, severe, and proliferative), and sight-threatening DR (STDR), which in our model included all forms of DR except mild-DR. Different from other studies, we included moderate-DR within STDR because all those patients in our sample had an associated diabetic macular edema.

Statistical Methods

Data evaluation and analysis was carried out using SPSS 22.0 statistical software at a significance of p<0.05. We measured the screening performance of the study using a confusion matrix/contingency. Given a classified data set, there were four basic combinations of actual and assigned: correct positive assignments, or true positives (TP), correct negative assignments, or true negatives (TN), incorrect positive assignments, or false positives (FP), and incorrect negative assignments, or false negatives (FN).

The statistical evaluation of the data set included:

sensitivity (S),

specificity (SP),

positive predictive value or precision (PPV),

negative predictive value (NPV),

positive false discovery rate or type 1 error (α) (FP),

negative false discovery rate or type 2 error (β) (FN), and

the accuracy or diagnostic effectiveness expressed as a proportion of correctly classified subjects (ACC).

ACC is affected by prevalence with the same S and SP, the diagnostic accuracy of a particular test increasing as the disease prevalence decreases, which was not the case in the present study. We did not use Area under the ROC curve (AUC) because it is a global measure of diagnostic accuracy but tells us nothing about individual parameters, such as S and SP. Where two tests have an identical or similar AUC, one can have significantly higher S and the other significantly higher SP. Furthermore, data on AUC says nothing about predictive values or about the contribution of the test in ruling-in and ruling-out a diagnosis. AUC can be determined when the classification method returns a continuous number between 0 and 1. Depending on a threshold, the answer can mean class 0 or class 1. Our method did not give a continuous value, but returned the class directly; therefore, no graph that depends on any threshold was suitable in this case.

Results

Demographic Study of Sample Size

The initial sample included 324 T1DM patients and 826 T2DM patients. Table 1 describes the demographic analysis of sample size.

Table 1

Demographic Characteristics of Our Sample Size

	T1DM Patients	T2DM Patients
Mean of current age	37.51 ± 9.85 years	51.44 ± 9.78 years
Sex male	168/324 (51.85%)	436/826 (52.78%)
DM duration	16.52 ± 8.59 years	15.98 ± 8.66 years
Mean of HbA1c	7.88 ± 0.94%	7.88 ± 0.94%

The prevalence of DR in the T1DM sample size was 81 (25%) patients who had any form of DR (25% of prevalence). By severity, 32 patients had mild-DR, 23 patients had moderate-DR, 21 patients had severe-DR and 5 patients had proliferative DR. Prevalence of DR sample size was 195 (23.60%) in T2DM patients and, by severity, 89 patients had mild-DR, 66 patients had moderate-DR, 31 patients had severe-DR and 9 patients had proliferative DR.

In the present study, sight-threatening diabetic retinopathy (STDR) was defined as all those who presented moderate-DR or higher. In total, STDR was diagnosed for 46 T1DM patients and 106 T2DM patients.

Trained Model with T1DM + T2DM Patients Applied to T1DM Patients

Table 2 shows the first trained model with T1DM together with T2DM. Results for predicting any-DR have poorer values than for predicting STDR. Predicting any-DR has ACC = 0.795 with S = 65.7%, which is low, and S = 65.7%. All these values are lower than those for predicting STDR with ACC= 0.918, S = 87.1% and a sensitivity of 87.8%. We conclude that this model is more effective for STDR than any-DR detection.

Table 2

The first algorithm applied to a sample of T1DM and T2DM patients and tested on a sample of T1DM and T2DM patients. The algorithm better estimated the risk of having more advanced forms of DR.

Study of Any-Diabetic Retinopathy
TP	FP	TN	FN	Error α	Accuracy0.795
64	65	318	33	0.17
Specificity	Sensitivity	PPV	NPV	Error β
83.0%	65.7%	49.6%	90.5%	0.35
Study of Sight-Threatening DR
TP	FP	TN	FN	Error α	Accuracy0.918
34	57	384	5	0.12
Specificity	Sensitivity	PPV	NPV	Error β
87.1%	87.8%	37.3%	98.7%	0.12

Trained Algorithm with Only T1DM Applied to T1DM Patients

Table 3 shows the second trained model with only T1DM patients. The prediction values for any-DR were ACC = 0.799, SP = 87.5%, and S = 86.3%, which were similar to those obtained in the first model but with higher sensitivity. However, this model proved to be highly effective in predicting STDR patients with an ACC = 0.937, SP = 90.9% and S = 83.0%. Comparing the two models, they were poor at predicting any-DR but effective at predicting STDR.Table 3. The second algorithm trained with only T1DM patients and tested on a sample of T1DM patients. It showed better results when predicting the risk of having sight-threatening DR compared to any form of DR.

Table 3

The second algorithm re-trained with only T1DM patients and tested on a sample of T1DM patients. It showed better results when predicting the risk of having sight-threatening DR compared to any form of DR.

Study of Any-Diabetic Retinopathy
TP	FP	TN	FN	Error α	Accuracy0.799
19	24	83	3	0.22
Specificity	Sensitivity	PPV	NPV	Error β
87.5%	86.3%	44.2%	96.5%	0.13
Study of Sight-Threatening DR
TP	FP	TN	FN	Error α	Accuracy0.937
10	20	98	1	0.09
Specificity	Sensitivity	PPV	NPV	Error β
90.9%	83.0%	33.3%	98.9%	0.17

Discussion

In a previous study, we demonstrated that the nine-year incidence of DR for T1DM patients was different from that found for T2DM patients. We were not able to show what the causes of these differences were based on the studied risk factors.8 In the present study, we wanted to determine whether our DR prediction model, which works well for patients with T2DM, was as effective in a sample of our patients with T1DM. Given that it is a population controlled by the same general practitioners and endocrinologists, we considered that the 9 independent variables used in our algorithm to determine levels of risk of DR might also be effective for patients with T1DM. However, the present study has only demonstrated that for risk of STDR and not for risk of any-DR.

As the developed algorithm was not effective enough, we decided to rework it (re-train) in two different ways, one with a mix of T1DM and T2DM patients then applying it to a similar sample, and another only with T1DM patients then applying that to a similar sample. However, neither of those two new algorithms worked effectively enough. The two models are effective at determining the risk for STDR but not for any-DR.In the first, we achieved high specificity (83%) to predict patients who will not develop DR but low sensitivity when predicting those at risk of developing DR. In the second, we achieved high specificity (86.3%) when predicting those who will not develop DR but a low sensitivity (65.7%) to predict T1DM patients at risk of developing DR when predicting those who will not develop. In addition, accuracy was not significant enough in either algorithm, 0.795 in the first and 0.799 in the second. We can conclude, therefore, that our algorithm is less effective for T1DM patients.

It is difficult to compare the present study to other similar studies since our results are poor. So far, two algorithms have proven useful in predicting DR in patients with T2DM, one developed by Aspelund et al,15 a second developed by Schreur21 and other by Broadbent et al,16,17 although the three studies predict only the risk of developing STDR and not any-DR.

Aspelund15 divided the patients according to diabetes type, T1DM or T2DM, which suggests that they had similar problems to ours for patients with T1DM. Their algorithm was based on the duration of DM, the HbA1c% value and systolic blood pressure values, independent variables that we also included, although their HbA1c values were fixed at 1% increases from 6% to 12%, whereas ours were not. In addition, we included renal function (microalbuminuria and glomerular filtration rate) and body mass index (BMI) values. The algorithm yielded good results for identifying those patients who might develop STDR,18–20 but not those who might develop the early stages of DR.

In a series of 268 patients with T1DM, after a follow-up of 4.6 years, Schreur et al,21 using Aspelund model demonstrated that the annual screening could be reduced by 58% if the condition of the retina initially is included as an independent variable within the algorithm, although, again, it was still only effective for patients at risk of STDR.

In the Broadbent et al model,16 the predictive results for patients with T2DM at risk of developing STDR are excellent. The algorithm is based on the following risk factors: age, sex, DM duration, systolic and diastolic pressure, HbA1c, and total cholesterol. It was applied to 179 patients with T1DM, but their results did not report the cohort by type of DM, so we do not know if they were as effective in the case of patients with T1DM as in the rest of the sample, which included a total of 4538 patients with DM.17,22

In summary, comparing the results of our study with these two models, we can say that the DR risk algorithms predict with some accuracy the risk of STDR in T1DM patients as our study, but not the risk of the early stages of DR (any-DR).

It is important to determine the risk of DR in its initial phase, because with a stricter glycaemic and arterial hypertension control, DR evolution can be delayed or its severity reduced, as evidenced by various studies.23–26 It is therefore important to continue to try to determine more accurately which DR risk factors will differentiate patients with T1DM or T2DM, so that we can predict risk in the initial stages rather than having to wait for STDR to appear.One limitation of the present study is the small sample size of 324 patients, which might not be sufficient to demonstrate our results with confidence. In addition, we limited ourselves to determining the effect of the 9 risk factors upon which our algorithm was built, therefore for T1DM patients insulin treatment might be useful as an additional variable in future studies. In addition, arterial hypertension is rare in patients with T1DM, and higher BMI scores might be more significant in T2DM patients.

Conclusions

The built model is useful for predicting risk of STDR but has limitations for predicting the risk of any-DR. It would be necessary to obtain a larger sample of patients to achieve better results.