Artificial Intelligence Based Machine Learning Models Predict Sperm Parameter Upgrading after Varicocele Repair: A Multi-Institutional Analysis.

PURPOSE: Varicocele repair is recommended in the presence of a clinical varicocele together with at least one abnormal semen parameter, and male infertility. Unfortunately, up to 50% of men who meet criteria for repair will not see meaningful benefit in outcomes despite successful treatment. We developed an artificial intelligence (AI) model to predict which men with varicocele will benefit from treatment.
MATERIALS AND METHODS: We identified men with infertility, clinical varicocele, and at least one abnormal semen parameter from two large urology centers in North America (Miami and Toronto) between 2006 and 2020. We collected pre and post-operative clinical and hormonal data following treatment. Clinical upgrading was defined as an increase in sperm concentration that would allow a couple to access previously unavailable reproductive options. The tiers used for upgrading were: 1-5 million/mL (ICSI/IVF), 5-15 million/mL (IUI) and >15 million/mL (natural conception). Thus moving from ICSI/IVF to IUI, or from IUI to natural conception, would be considered an upgrade. AI models were trained and tested using R to predict which patients were likely to upgrade after surgery. The model sorted men into categories that defined how likely they were to upgrade after surgery (likely, equivocal, and unlikely).
RESULTS: Data from 240 men were included from both centers. A total of 45.6% of men experienced an upgrade in sperm concentration following surgery, 48.1% did not change, and 6.3% downgraded. The data from Miami were used to create a random forest model for predicting upgrade in sperm concentration. On external validation using Toronto data, the model accurately predicted upgrade in 87% of men deemed likely to improve, and in 49% and 36% of men who were equivocal and unlikely to improve, respectively. Overall, the personalized prediction for patients in the validation cohort was accurate (AUC 0.72).
CONCLUSIONS: A machine learning model performed well in predicting clinically meaningful post-varicocelectomy sperm parameters using pre-operative hormonal, clinical, and semen analysis data. To our knowledge, this is the first prediction model to show the utility of hormonal data, as well as the first to use machine learning models to predict clinically meaningful upgrading. This model will be published online as a clinical calculator that can be used in the preoperative counseling of patients.

INTRODUCTION

Varicocele is the most common correctable cause of infertility in men, with varicocelectomy being the most common surgery performed for male factor infertility [12]. Factors that have been shown to improve following treatment for a clinically apparent varicocele include DFI [3], assisted reproduction technology (ART) success [4], natural pregnancy rate [5], semen parameters [15], and possibly testosterone [6]. Despite this, one of the challenges of this surgery is accurate patient selection, as several series report that a significant proportion of men fail to respond with improved semen parameters [78]. Being able to predict the chance of improved semen parameters will avoid both unnecessary treatment and potential delay for ART.

Traditionally, varicocele success rate, defined as improvement in semen parameters, is between 66% to 70.7% [91011]. Recently, a push to quantify success as a “clinically meaningful” change in semen parameters has emerged [12]. A clinically meaningful change is one in which semen parameters improve enough to allow access to a previously unavailable form of reproduction. For couples, this would mean upgrading from in vitro fertilization (IVF) as the only available option, to one where the couple could now attempt intrauterine insemination (IUI), or moving from IUI to natural conception. When defining success of varicocelectomy using sperm parameter upgrading in this way, up to 50% of men fail to benefit from varicocele surgery [7].

In an attempt to better select men for this procedure, a nomogram using preoperative variables has been created [13]. However, this nomogram does not perform as well on multicenter validation attempts, calling into question its utility in the preoperative counseling of men [14]. Additionally, no prediction models currently exist that incorporate hormonal data, which have been shown to influence the potential for successful repair [81516]. Finally, it is important for prediction models to use clinically meaningful upgrading as an outcome, as a slight change in sperm concentration, while statistically significant, may not change the reproductive options available to the couple.

Recently, artificial intelligence (AI) models have been used in urology to help model and predict outcomes, in many cases with better success than with traditional nomograms [1718].

Our objective is to use AI to predict meaningful clinical outcomes after varicocele treatment using an externally validated, multicenter, international dataset. We hypothesize that using an AI model will accurately predict likelihood of sperm concentration upgrade after varicocelectomy.

MATERIALS AND METHODS

Data was collected from a prospectively maintained database at each center, collected from 2006 to 2020. Men presenting with infertility (defined as 1 year of unsuccessful attempts at conception), a palpable varicocele, and at least one abnormal semen parameter, who underwent surgery or embolization were included. Men with azoospermia or cryptozoospermia were excluded. Only men who had treatment for the purposes of improving fertility outcomes were included. Men were excluded if they were on hormonal therapies (selective estrogen receptor modulators, aromatase inhibitors etc) at the time of semen analyses. All examinations were performed by male infertility specialists (RR and KJ) with fellowship training. Subclinical varicoceles were not treated.

Clinical and demographic data was extracted from both databases (Table 1). Extracted data included age, varicocele laterality, surgery xlink:type, testis volume, varicocele grade, as well as pre-operative and post-operative semen analysis parameters, follicle stimulating hormone (FSH), and testosterone. For all men, the post-operative semen analysis was performed between 3 and 9 months after treatment. Semen analyses were analyzed in keeping with 2010 World Health Organization (WHO) guidelines [19]. Missing clinical data accounting for <15% of each variable, were imputed through the median. Men were abstinent for 2 to 5 days prior to depositing the semen analysis. Surgery xlink:type was either microscopic subinguinal varicocelectomy, performed by male infertility specialists (RR and KJ) with fellowship training, or radiographic embolization by a fellowship-trained interventional radiologist. Microsurgical embolization was performed under general anesthetic, using a subinguinal incision. Using an operating microscope, all veins were either tied or clipped, taking care to preserve the testicular artery through use of a microdoppler, and taking care to preserve at least one lymphatic. Radiographic embolization was performed by getting vascular access through the right common femoral vein. Venography is performed, after which embolization is achieved using either mechanical coils or liquid embolic agents. A small number of right-sided grade 0 varicoceles were treated with embolization at the time of a planned left clinical varicocele embolization. This was done based on the presence of reflux on diagnostic venogram. Treatment choice of surgery or embolization was selected by the patient in a shared-decision making approach.

Table 1

Baseline clinical data

Variable		Miami, USA	Toronto, Canada	p-value
Number of patients		160	80
Age (y)		35 (30–39)	37 (32.0–39.3)	0.078
Varicocele laterality (%)				0.025
	Unilateral	88 (55.0)	31 (38.7)
	Bilateral	72 (45.0)	49 (61.3)
Left grade				0.624
	1	36 (22.5)	19 (23.8)
	2	64 (40.0)	36 (45.0)
	3	60 (37.5)	25 (31.2)
Right grade				0.012
	0	0 (0.0)	5 (10.2)
	1	30 (41.1)	30 (61.2)
	2	36 (49.3)	13 (26.5)
	3	7 (9.6)	1 (2.0)
Baseline semen analysis
	Volume (mL)	2.65 (1.80–3.60)	2.75 (2.0–4.0)	0.59
	Concentration (million/mL)	5.45 (2.0–12.0)	7.68 (3.95–12.85)	0.01
	Motility %	29 (16.0–47.0)	19 (11.88–26.06)	<0.001
	TMSCa	3.56 (0.65–10.25)	3.67 (1.78–6.35)	0.83
Baseline FSH (mIU/mL)		6.4 (4.2–9.4)	6.4 (3.9–10.0)	0.95
Baseline testosterone (ng/dL)		437 (317.75–555.00)	403 (310–490)	0.07
Change after varicocele surgery
	None	77 (48.1)	35 (43.8)
	Upgrade	73 (45.6)	39 (48.8)
	Downgrade	10 (6.3)	6 (7.5)

Values are presented as number only, median (interquartile range), or number (%).

TMSC: total motile sperm count, FSH: follicle stimulating hormone.

aTMSC=volume×concentration×motility.

p<0.05 is deemed significant.

Outcome measures were based on sperm concentration in an effort to compare the most straightforward yet meaningful parameter across centers. In comparison between labs, measurement of sperm concentration appears to vary the least, when compared with motility and vitality [20]. Men were pre-sorted into categories based on sperm concentration levels that have been used previously to determine which forms of ART are available to couples: 1–5 million (IVF), 5–15 million (IUI), and >15 million (natural conception) [2122]. Men were deemed to have upgraded if sperm concentration increased enough to enter into a new category. Therefore, our primary outcome was a clinically meaningful change in sperm concentration.

1. Prediction modeling

University of Miami data was used for model development. After determination of clinically meaningful improvement in sperm concentration after varicocelectomy, medians and interquartile ranges (IQRs, 25%–75%) of independent variables were reported based on lab reference ranges. Multivariable-adjusted logistic regression analysis was performed to determine odds of upgrade. A p-value <0.05 was considered statistically significant. These data were used to inform model parameters. Model inputs were surgical laterality (unilateral or bilateral), baseline semen concentration and FSH. FSH was divided into low, normal, and high (1.5–12.4 mIU/mL) for the regression analysis, and analyzed as a continuous variable for the AI modeling. Three supervised machine learning candidate models were evaluated: logistic regression, support vector machine and random forest. Models were initially internally validated using the same Miami dataset. A random forest model was selected as it had the best discriminatory performance (Fig. 1). The random forest outputs a numerical value between 0 and 1. Cut-off thresholds, above and below which predict clinical upgrade or not, were selected based upon internal validation with Miami data. It was determined from these data that there were three roughly evenly sized groups: those with low, equivocal or high likelihood of clinically significant sperm concentration upgrade.

Fig. 1

Calibration curve for post-repair sperm concentration. This curve measures the correspondence between the AI-predicted model and the actual observed values. The grey line represents the ideal scenario when predicted values equal observed values. The dotted line is the calibration curve of the AI model. The model shows good calibration. AI: artificial intelligence.

University of Toronto data were used for external validation of the random forest model. Institutional medians and IQRs (25%–75%) of independent variables were determined to compare the two datasets. Model performance was assessed by accuracy of low, equivocal or high likelihood upgrade groupings, discrimination area under the receiving operating curve (AUC), and calibration. Calibration plots, using cross-validation, define how well the predicted probabilities match the actual probability of sperm concentration upgrade. The random forest model was run twelve times. All model characteristics are the mean of ten model runs with the highest and lowest preforming runs removed. All statistical analyses were performed using R Statistical Software.

2. Ethics statement

Approval of Institutional Review Board was obtained at the University of Miami (approval number: 20170849) and at the University of Toronto (approval number: 14-0342-E), respectively. Written informed consent was not required from patients. This was a retrospective study, and so deidentified data was used.

RESULTS

A total of 240 men were included in the final analysis, 160 men from the University of Miami and 80 men from the University of Toronto. Median age was 36 years (IQR, 31–39 y). Fifty percent of men underwent unilateral varicocelectomy. The most common grade for varicocele repaired on the left was grade 2 (n=100; 41.7%). The most common varicocele grade observed on the right was grade 1 (n=60, 49.2%). Average baseline semen concentration was 4.5 million/mL (IQR, 3–12 million/mL). Average baseline total motile sperm count (TMSC) was 3.59 (IQR 0.9-8). Average baseline FSH and testosterone were 6.4 mIU/mL (reference range, 1.5–12.4 mIU/mL) and 420 ng/dL (reference range, 300–900 ng/dL), respectively. Average testis volume was 14 mL (IQR, 12–16 mL). See Table 1 for details.

1. Model development data set

From the University of Miami, complete datasets were available from 219 men. After applying exclusion criteria removing men with azoospermia and cryptozoospermia (n=20; 9.1%) and men with a baseline sperm concentration greater than 15 million/cc (n=39; 17.8%) data were available from 160 (73.1%) men. Of these men, 73 (45.6%) had a clinically meaningful upgrade in sperm concentration after varicocelectomy. The clinical characteristics of men in the Miami cohort who upgraded semen parameters and those who did not are shown in Table 2. Men with high FSH and bilateral repair had a lower odds of clinically meaningful sperm concentration upgrade after varicocelectomy, odds ratio (OR) 0.2 (95% confidence interval [CI] 0.07–0.59; p=0.003) and OR 0.47 (CI 0.24–0.94; p=0.03), respectively. A random forest model to predict low, equivocal or high likelihood of clinically meaningful sperm concentration upgrade was developed from these data.

Table 2

Characteristics of men who upgraded and men who did not upgrade from University of Miami data

Variable		Upgrade	Not upgrade	p-value
Number of patients		73	77
Age (y)		34 (30–38)	35 (29–39)	0.62
Varicocele laterality (%)				0.047
	Unilateral	47 (64.4)	38 (49.4)
	Bilateral	26 (35.6)	39 (50.6)
Left grade				0.52
	1	16 (21.9)	19 (24.7)
	2	26 (35.6)	32 (41.6)
	3	31 (42.5)	26 (33.8)
Right grade				0.051
	0	47 (64.4)	37 (48.1)
	1	9 (12.3)	19 (24.7)
	2	15 (20.5)	18 (23.4)
	3	2 (2.7)	3 (3.9)
Baseline semen analysis
	Volume (mL)	2.8 (2.0–3.8)	2.5 (1.7–3.5)	0.29
	Concentration (million/mL)	4.0 (2.0–12.0)	6.0 (0.8–11.0)	0.28
	Motility %	29.0 (16.0–44.0)	31.0 (15.8–48.0)	0.57
	TMSC	3.53 (0.83–10.62)	3.42 (0.42–10.13)	0.64
Baseline FSH (mIU/mL)		6.2 (3.7–8.4)	6.6 (4.4–12.8)	0.28
Baseline testosterone (ng/dL)		437 (317–558)	438 (322–548)	0.96

Values are presented as number only, median (interquartile range), or number (%).

TMSC: total motile sperm count, FSH: follicle stimulating hormone.

p<0.05 is deemed significant.

2. Model validation data set

From the University of Toronto, complete datasets including baseline FSH and pre- and post-varicocelectomy semen analysis were available from 142 men. After applying exclusion criteria removing men with azoospermia (n=5; 3.5%) and men with a baseline sperm concentration greater than 15 million/mL (n=57; 40.1%) data were available from 80 (56.3%) men. Of these men, 39 (48.8%) had a clinically meaningful upgrade in sperm concentration after varicocelectomy. Eleven percent (n=26) underwent radiological embolization, whereas the remaining men (n=214; 89%) underwent microscopic subinguinal varicocelectomy. No men in the Miami cohort underwent embolization. The clinical characteristics of men in the Toronto cohort who upgraded and those who did not are shown in Table 3.

Table 3

Characteristics of men who upgraded and men who did not upgrade from University of Toronto data

Variable		Upgrade	Not upgrade	p-value
Number of patients		39	35
Age (y)		36 (32–39)	37 (32–39)	0.96
Varicocele laterality (%)				0.95
	Unilateral	15 (38.5)	14 (40.0)
	Bilateral	24 (61.5)	21 (60.0)
Left grade				0.009
	1	7 (17.9)	8 (22.9)
	2	23 (59.0)	11 (31.4)
	3	9 (23.1)	16 (45.7)
Right grade				0.89
	0	19 (48.7)	15 (42.9)
	1	14 (35.9)	13 (37.1)
	2	5 (12.8)	7 (20.0)
	3	1 (2.6)	0 (0.0)
Baseline semen analysis
	Volume (mL)	3.0 (2.0–4.0)	2.50 (1.98–3.38)	0.32
	Concentration (million/mL)	5.0 (3.2–10.5)	10.9 (5.7–18.0)	0.005
	Motility %	21.0 (15.5–29.5)	13.4 (8.1–21.1)	0.003
	TMSC	3.96 (1.39–6.33)	3.02 (1.75–6.00)	0.58
Baseline FSH (mIU/mL)		6.5 (3.9–11.0)	6.9 (4.0–9.1)	0.82
Baseline testosterone (ng/dL)		346.1 (288.4–447.0)	403.7 (317.2–490.3)	0.35

Values are presented as number only, median (interquartile range), or number (%).

TMSC: total motile sperm count, FSH: follicle stimulating hormone.

p<0.05 is deemed significant.

3. Model performance

On external validation, a random forest supervised machine learning model accurately determined whether each man had a low, equivocal or high likelihood of clinically meaningful semen concentration upgrade after varicocelectomy. Men who were designated by the model as highly likely to upgrade did upgrade 86.7% (range, 81.2%–92.3%) of the time. The model predicted upgrading from IVF to IUI and IUI to natural conception in equivalent frequency. On average 18.8% (15/80) men were designated to be highly likely to upgrade. Men designated as equivocal upgraded 48.7% (38.5%–57.1%) of the time. On average 22.5% (18/80) men were designated as equivocal. Men with a low likelihood of upgrade designation upgraded 36.4% (30.8%–39.1%) of the time. On average 58.8% (47/80) men were designated to be low likelihood of upgrade. Table 4 outlines the accuracy of prediction in each of the three groups (highly likely, equivocal, and unlikely) compared to the real-world data. The model performed with good discriminatory ability with an AUC of 0.72 (0.71–0.73) and with good calibration (Fig. 1). The characteristics of the men most likely and least likely to upgrade are outlined in Table 5.

Table 4

External validation predicted versus actual upgrade after varicocelectomy

Real-world results	AI model prediction
	Likely to upgrade (n=15)	Equivocal (n=18)	Unlikely to upgrade (n=47)
Upgrade (n=39)	13	9	17
Not upgraded (n=35)	2	8	25
Downgrade (n=6)	0	1	5

These data show the model is specific for accurately predicting which men are likely to upgrade.

Table 5

Data from the 15 men who were least and most likely to upgrade

Group	FSH (median, mIU/mL)	Concentration (median, million/mL)	Laterality (% B/L)
Lowest likelihood	12.45	5.15	62.50
Highest likelihood	5.15	2.10	7.14

FSH: follicle stimulating hormone, B/L: bilateral.

DISCUSSION

Varicocele repair is recommended in all infertile men with a clinical varicocele and at least one abnormal semen parameter according to recent AUA guidelines [2]. Following these guidelines will result in some men having unnecessary treatment, as many men fail to improve following surgery [7], a phenomenon seen in this series as well. In this study, we developed an AI-based machine learning model to help predict which men would experience clinically meaningful improvement in sperm concentration. We incorporated hormonal data which is novel amongst prediction models, and externally validated our model with a dataset from another high-volume center. We found that a random forest AI model predicted clinically significant upgrading with accuracy that exceeded that of logistic regression. By using multiple predictive thresholds in the model, we were able to separate men into more clinically useful categories and thus advise men if they have a high, equivocal, or low likelihood of upgrading. The results of this research and the resulting prediction tool can help clinicians counsel men in a meaningful way to help them better understand their chances for success when undergoing varicocele repair. These results can be easily translated into a web-based calculator, which we plan to publish and make available for clinical use.

One novel factor in our model was incorporation of serum FSH. We found that men with an elevated FSH were less likely to experience an upgrade. This is supported by findings in other individual series [823]. Additionally, a recent meta-analysis found that serum FSH decreases following surgery, possibly due to improvement in spermatogenesis [15]. Utilizing FSH in predicting surgical outcomes has already been incorporated into Urologists’ practice, and based on our model should be added to the initial diagnostic workup of men with varicocele [24].

Our study has both strengths and limitations. To our knowledge, we believe we are the first to use an AI, machine learning model to predict who will improve sperm parameters following varicocele repair. Additional strengths of this study include its multi-institutional, multi-surgeon, international cohort, which increases the generalizability and validity of our results. We established outcomes based on clinically meaningful changes to sperm parameters, which provides translatable options to couples during preoperative counseling. The predictive accuracy of our model is robust with an AUC of 0.72. For context, prostate-specific antigen test has an AUC of between 0.68 and 0.66 [25], the memorial Sloan Kettering prostatectomy nomogram has an AUC of 0.74 [26], the R.E.N.A.L nephrometry score for predicting malignancy is 0.76 [27]. Our model’s accuracy is similar in nature to tests used commonly in other fields within urology and thus has merit for routine clinical use. Limitations of our study include the lack of detailed demographic data, as several of these factors (age, BMI, smoking) have been shown to further reduce semen quality in men diagnosed with varicoceles [28]. Additionally, TMSC is increasingly being used to predict which reproductive options are available to couples, which was not used in this study [712]. Because of our lack of lab centralization between the centers, we opted to use sperm concentration due to its simplicity and minimal variability between labs [20]. Additionally, external validation attempts of nomograms using TMSC have shown incongruent correlation coefficients, indicating a potential for lab variations in motility calculation [14]. We did run the model using TMSC as our main outcome measure as an exploratory outcome, however this yielded a poor AUC of 0.54, possibly giving credence to differing motility evaluations between labs. In our series, 45.6% of men had a clinically significant upgrade in sperm concentration. This is similar in nature to other studies that have assessed some form of clinically significant upgrading, which ranges between 50% and 58% [712]. Our finding may be lower due to the inclusion of embolization, which has been shown to have a higher failure rate when compared to subinguinal microscopic varicocelectomy [5].

CONCLUSIONS

In this study, we used an AI, machine-learning model to predict semen parameter upgrading following varicocele repair. We used hormonal, clinical and sperm data to predict clinically significant improvements in semen parameters in men trying to conceive. Our model displayed superior performance when compared to a more traditional, logistic regression. The model highlighted the importance of FSH and bilaterality in predicting improvement, and will be published as an online calculator for clinicians to use. Models such as these can be used in clinic to assist with decision making before varicocele repair, and to help manage expectations afterwards.