Development and Validation of Dynamic Multivariate Prediction Models of Sexual Function Recovery in Patients with Prostate Cancer Undergoing Radical Prostatectomy: Results from the MUSIC Statewide Collaborative.

Background: Radical prostatectomy (RP) is the most common definitive treatment for men with intermediate-risk prostate cancer and is frequently complicated by erectile dysfunction. Objective: To develop and validate models to predict 12- and 24-month post-RP sexual function. Design setting and participants: Using Michigan Urological Surgery Improvement Collaborative (MUSIC) registry data from 2016 to 2021, we developed dynamic, multivariate, random-forest models to predict sexual function recovery following RP. Model factors (established a priori) included baseline patient characteristics and repeated assessments of sexual satisfaction, and Expanded Prostate Cancer Index Composite 26 (EPIC-26) overall scores and sexual domain questions. Outcome measurements and statistical analysis: We evaluated three outcomes related to sexual function: (1) the EPIC-26 sexual domain score (range 0-100); (2) the EPIC-26 sexual domain score dichotomized at ≥73 for "good" function; and (3) a dichotomized variable for erection quality at 12 and 24 months after RP. A gradient-boosting decision tree was used for the prediction models, which combines many decision trees into a single model. We evaluated the performance of our model using the root mean squared error (RMSE) and mean absolute error (MAE) for the EPIC-26 score as a continuous variable, and the area under the receiver operating characteristic curve (AUC) for the dichotomized EPIC-26 sexual domain score (SDS) and erection quality. All analyses were conducted using R v3.6.3. Results and limitations: We identified 3983 patients at 12 months and 2494 patients at 24 months who were randomized to the derivation cohort at 12 and 24 months, respectively. Using baseline information only, our model predicted the 12-month EPIC-26 SDS with RMSE of 24 and MAE of 20. The AUC for predicting EPIC-26 SDS ≥73 (a previously published threshold) was 0.82. Our model predicted 24-month EPIC-26 SDS with RMSE of 26 and MAE of 21, and AUC for SDS ≥73 of 0.81. Inclusion of post-RP data improved the AUC to 0.91 and 0.94 at 12 and 24 months, respectively. A web tool has also been developed and is available at https://ml4lhs.shinyapps.io/askmusic_prostate_pro/. Conclusions: Our model provides a valid way to predict sexual function recovery at 12 and 24 months after RP. With this dynamic, multivariate (multiple outcomes) model, accurate predictions can be made for decision-making and during survivorship, which may reduce decision regret. Patient summary: Our prediction model allows patients considering prostate cancer surgery to understand their probability before and after surgery of recovering their erectile function and may reduce decision regret.

Introduction

Radical prostatectomy (RP) is the treatment most frequently selected for localized prostate cancer and is curative for early-stage prostate cancer [1], [2]. Although effective from the perspective of cancer control, RP has nontrivial potential morbidity. Erectile dysfunction impacts many patients undergoing RP [3] and is highly correlated with patient satisfaction and decision regret [4], [5]. While survival after prostate cancer is the key outcome, some men may be willing to sacrifice survival to remain potent [6]. In order to ensure that patients have gains in both quantity and quality of life, sexual function outcomes must not be neglected in the shared decision-making process.

Prevalence estimates of erectile dysfunction after RP from population-based studies widely vary and are often the only data available for counseling patients, but they do not accurately represent every individual’s erectile function trajectory following prostate cancer treatment [7], [8], [9], [10]. Compounding this is the fact that patients have poor ability to anticipate changes in erectile function following RP and may have overly optimistic expectations [11], [12]. A major limitation in decision-making for individuals contemplating treatment for prostate cancer is the paucity of studies that use contemporary data from patients treated in diverse practices for individualized prediction of their own outcomes before and after RP. As patients with prostate cancer face the reality of their erectile function after surgery, it is also important to be able to provide ongoing counseling based on real-time (ie, postoperative) data to provide realistic expectations, which may decrease decision regret and facilitate treatment decision-making based on patient preferences. There is a critical need for tools to predict post-RP erectile function for perioperative patient counseling, shared decision-making, high-quality and high-value care, and cancer survivorship.

Existing models for predicting functional outcomes following prostate cancer surgery have demonstrated good performance but are only intended for preoperative counseling and do not examine other domains of sexuality, such as sexual satisfaction [13], [14], [15], [16]. Here we build on that work in predicting erectile function using novel predictors, preoperative data, and postoperative data in a “dynamic” prediction model that may have high utility for prostate cancer survivorship. In this context, we used data from the Michigan Urological Surgery Improvement Collaborative (MUSIC) clinical registry to determine whether routinely collected clinical variables can reliably predict erectile function outcomes using machine learning. We specifically use 2-yr longitudinal data on sexual function to develop an accurate, valid, and reliable method for predicting sexual function recovery on the basis of patient-level determinants. Our findings will provide patients with the ability to make informed decisions and give clinicians an evidence-based tool to facilitate shared decision-making and postoperative counseling, which could potentially help in reducing decision regret.

Patients and methods

Data source

MUSIC is a physician-led quality improvement collaborative established in 2011 in partnership with Blue Cross Blue Shield of Michigan [17]. The collaborative comprises 46 diverse community and academic urology practices representing approximately 90% of the urologists in the state. A major objective of MUSIC is to improve patient-reported outcomes after RP. MUSIC Patient Reported Outcomes (MUSIC-PRO) is a statewide web-based system established in 2014 for measuring and improving outcomes and health-related quality of life (HRQOL) following RP, with data collection at baseline (before surgery) and at 3, 6, 12, and 24 months after surgery [18]. Prior reports have described the data acquisition and quality control activities in MUSIC [19], [20]. Each MUSIC practice obtained an exemption or approval for collaborative participation from a local institutional review board.

Study cohort

The cohort in this study included all men in the MUSIC registry who underwent RP from February 2016 to January 2021 and enrolled in MUSIC-PRO. We then divided patients into derivation and validation cohorts using 2:1 random sampling of the overall cohort stratified by surgeon. Thus, each surgeon was represented in both the derivation and validation cohorts. However, no patient was included in both cohorts.

Measures

MUSIC-PRO

The MUSIC-PRO survey administered from May 2016 onwards uses the Expanded Prostate Cancer Index Composite-26 (EPIC-26) instrument [21]. Single-item measures from the Patient-Reported Outcome Measurement Information System (PROMIS) Interest in Sexual Activity and Global Satisfaction with Sex Life subdomains [22], [23] are also included in the survey. The surveys were administered at baseline (before RP) and at 3, 6, 12, and 24 months after RP.

MUSIC registry

The MUSIC registry includes detailed clinical and demographic information including patient age, race, prostate specific antigen (PSA), comorbidities, Gleason score, T stage, BMI, nerve sparing (none, unilateral, and bilateral), surgeon, and MUSIC-PRO survey responses.

Statistical analysis

Predictors

Predictors were established a priori and selected on the basis of prior studies [7], [23], [24]. The a priori predictors included age, body mass index, diabetes, PSA, Gleason grade, prostate volume, EPIC-26 sexual domain questions and overall score, and the PROMIS Satisfaction with Sex Life single item measure. The EPIC-26 sexual domain questions and PROMIS satisfaction were assessed at baseline (before RP) and at 3, 6, 12, and 24 months after RP. A complete list of predictors is included in Supplementary Table 1.

Outcomes

We evaluated three outcomes related to sexual function: (1) the EPIC-26 sexual domain score (range 0–100); (2) the EPIC-26 sexual domain score dichotomized at ≥73 for “good” function; and (3) a dichotomized erection quality variable at 12 and 24 months following RP. The threshold of ≥73 for dichotomizing EPIC-26 scores was chosen on the basis of prior work [25], [26]. Good erection quality was defined as “erections firm enough for intercourse”.

Model development

Gradient-boosting decision tree (GBDT) models were used for the prediction model, in which many decision trees are combined into a single model. Predictions made by combining decision trees together results in greater accuracy. We constructed two GBDT models: one to predict EPIC-26 outcomes at 12 and 24 months after RP, and one to predict erection quality at 12 and 24 months after RP. We refer to the models as dynamic because they can make 12-month and 24-month predictions at different time points using continuously collected data beyond the baseline/preoperative time point (e.g., at baseline/preoperative, 3 months/postoperative), and as multivariate because they predict outcomes at multiple time points (e.g., 12 and 24 months after RP). Uncertainty in the EPIC-26 outcomes was estimated using GBDTs trained to predict the 2.5th and 97.5th percentiles for EPIC-26 scores.

Model validation

For the EPIC-26 sexual domain score, we evaluated the performance of our model using the root mean squared error (RMSE) and mean absolute error (MAE). We used the area under the receiver operating characteristic curve (AUC) to evaluate discrimination of dichotomized EPIC-26 sexual domain scores and erection quality. We assessed calibration by visually comparing predicted values and observed scores using calibration plots.

Importance of variables

We evaluated the importance of variable (i.e., which predictors were most predictive) by summing the relative influence of each predictor across all trees [27].

Sensitivity analyses

Three additional predictors were added to examine changes in model performance: surgical volume, nerve-sparing status, and use of erectile aids.

All analyses for the current study were conducted with R v3.6.3, with models fitted using the h2o package and calibration plots generated using the runway package for R.

Results

Sample description

We identified 3983 men with EPIC-26 sexual domain scores available at 12 months and 2494 with scores available at 24 months. At 12 months, 2653 men were randomized to the derivation cohort and 1657 to the validation cohort (Table 1). At 24 months, 1330 men were randomized to the derivation cohort and 837 to the validation cohort. The mean age of the cohort was 64 yr in the derivation cohort and 65 yr in the validation cohort. The median age was 65 yr in both the derivation cohort and the validation cohort. The derivation cohort was further randomly subdivided into training and tuning cohorts, with the tuning cohort used to determine when the model had converged on the basis of early stopping criteria.

Table 1

Patient characteristics for predictors and outcomes by cohort and outcome time

Characteristic a	Derivation cohort		Validation cohort
	12-mo outcome(N = 2653)	24-mo outcome(N = 1657)	12-mo outcome(N = 1330)	24-mo outcome(N = 837)
Age (yr)	65 (60–69)	65 (60–69)	65 (60–69)	65 (60–69)
Body mass index (kg/m2)	28.7 (26.1–32.2)	28.6 (25.9–32.3)	28.7 (26.2–32.1)	28.4 (26.1–31.8)
Data missing	538	363	274	179
Diabetes mellitus, n (%)	200 (9.4)	121 (9.2)	118 (11)	63 (9.5)
Data missing	515	348	262	172
Prostate volume (cm3)	37 (28–49)	37 (28–50)	36 (28–49)	36 (29–48)
Data missing	824	469	461	297
Prostate-specific antigen (ng/ml)	6.5 (4.8–9.4)	6.4 (4.7–9.3)	6.3 (4.8–9.5)	6.2 (4.8–9.8)
Data missing	824	469	403	237
Gleason grade group, n (%)
Grade group 1	343 (15)	231 (16)	168 (14)	126 (16)
Grade group 2	1,012 (44)	632 (43)	557 (47)	345 (45)
Grade group 3	532 (23)	319 (22)	261 (22)	159 (21)
Grade group 4	265 (11)	174 (12)	123 (10)	82 (11)
Grade group 5	171 (7.6)	114 (7.8)	79 (6.6)	54 (7.0)
Data missing	330	187	142	71
Sexual satisfaction at baseline, n (%)
1	40 (2.7)	18 (1.8)	21 (2.9)	9 (1.9)
2	128 (8.6)	82 (8.4)	58 (8.0)	31 (6.6)
3	347 (23)	237 (24)	168 (23)	113 (24)
4	523 (35)	338 (35)	248 (34)	160 (34)
5	455 (30)	304 (31)	230 (32)	157 (33)
Data missing	1160	678	605	367
EPIC-26 SDS at baseline	71 (40–88)	72 (42–91)	71 (40–92)	75 (46–92)
Data missing	537	267	263	129
EPIC-26 SDS after RP	22 (10–50)	25 (12–57)	22 (9.0–53)	26 (12–62)
High erection quality after RP, n (%)	412 (16)	318 (19)	203 (15)	178 (21)
Data missing	3	0	0	0

EPIC-26 = Expanded Prostate Cancer Index Composite-26; RP = radical prostatectomy; SDS = sexual domain score.

Data for continuous variables are presented as the median (interquartile range).

Prevalence of outcomes

EPIC-26 sexual domain score

At baseline, the median EPIC-26 sexual domain score was 71 (IQR 40–89); 48% of patients had a good domain score, defined as a score ≥73. The median sexual domain score was 22 (IQR 9.5–51) at 12 months after RP and 26 (IQR 13–58) at 24 months, with good domain scores reported by 13% and 16%, respectively.

Erection quality

Good erection quality was reported by 56% at baseline (1803/3239), 15% (615/3980) of the overall cohort at 12 months after RP, and 19% (496/2494) at 24 months after RP.

Model performance in the validation cohort

EPIC-26 sexual domain score at 12 months

Using baseline information only, our model predicted the EPIC-26 sexual domain score at 12 months with RMSE of 24 and MAE of 20 (Table 2). The AUC for predicting a sexual domain score ≥73 was 0.82 (Table 2 and Supplementary Fig. 1). On inclusion of post-RP data at 3 and 6 months, RMSE improved to 17, MAE improved to 13, and AUC improved to 0.91.

Table 2

Model performance for the EPIC-26 sexual domain score and erection quality

Outcome and time	Data for patient-reported predictors				Model performance
	BL	3 mo	6 mo	12 mo	RMSE	MAE	AUC
EPIC-26 sexual domain score
12 mo	X				24	20	0.82
		X			21	16	0.85
			X		19	14	0.89
	X	X	X		17	13	0.91
24 mo	X				26	21	0.81
		X			23	18	0.84
			X		21	16	0.88
				X	17	12	0.93
	X	X	X	X	17	13	0.94
Erection quality
12 mo	X						0.80
		X					0.82
			X				0.86
	X	X	X				0.89
24 mo	X						0.81
		X					0.85
			X				0.88
				X			0.92
	X	X	X	X			0.92

AUC = area under the receiver operating characteristic curve; BL = baseline; EPIC-26 = Expanded Prostate Cancer Index Composite-26; MAE = mean absolute error; RMSE = root mean squared error.

EPIC-26 sexual domain score at 24 months

Using baseline information only, our model predicted the EPIC-26 sexual domain score at 24 months with RMSE of 26 and MAE of 21, and an AUC for sexual domain score ≥73 of 0.81 (Table 2 and Supplementary Fig. 2). On inclusion of post-RP data for 3, 6, and 12 months, RMSE improved to 17, MAE improved to 12, and AUC improved to 0.94.

Erection quality at 12 months

Using baseline information only, our model predicted good erection quality at 12 months, with an AUC of 0.80, which improved to 0.89 on inclusion of post-RP data at 3 and 6 months (Table 2).

Erection quality at 24 months

Using baseline information only, our model predicted good erection quality at 24 months, with an AUC of 0.81, which improved to 0.92 on inclusion of post-RP data at 3 and 6 months (Table 2).

Model calibration

The model predicting the EPIC-26 sexual domain score was well calibrated at 12 and 24 months according to the fit of a local regression curve (Fig. 1). The model predicting erection quality appeared to be well calibrated at 12 months but underpredicted the probability of good erection quality at 24 months (Fig. 2).

Fig. 1

Calibration of the model predicting EPIC-26 sexual domain score for the validation cohort using baseline information only at 12 and 24 mo with locally weighted scatterplot smoothing and a density plot of predicted scores below.

Fig. 2

Calibration of the model predicting high erection quality at 12 and 24 mo for the validation cohort using baseline information only with locally weighted scatterplot smoothing and a density plot of predicted scores below.

Shapley value plots are provided in Supplementary Figs. 3–6 for the EPIC-26 sexual domain score and erection quality, with separate plots provided for factors important at baseline and at 6 mo after surgery. The two most important factors for predicting the 24-month EPIC-26 sexual domain score at baseline were the baseline sexual domain score and age. At 6 month after surgery, the most important factor was the 6-month sexual domain score, with age remaining as the second most important variable. The same variables were also the two most important variables for predicting good erectile quality, although age was the most important factor. We examined the impact of surgeon volume and noted it did not improve the model beyond patient-level determinants.

The model performance was unchanged or minimally improved on addition of predictors including surgical volume, nerve-sparing status, and erectile aids. For example, addition of surgical volume did not improve prediction of the EPIC-26 score at 24 months using baseline data (RMSE 26, MAE 21, AUC 0.81) but did slightly improve the RMSE (from 17 to 16) when predicting using complete data through 12 months. Inclusion of nerve-sparing status similarly improved the RMSE from 17 to 16 when using complete data through 12 months to predict 24-month outcomes.

For the impact of erectile aid use, we first examined the prevalence of erectile aid use and then investigated model performance on inclusion of erectile aids. At 12 months, 11.3% (n = 187) of patients reported use of a vacuum erection device, 7.9% (n = 131) reported use of intracavernosal injections, and 32.2% (n = 533) reported use of phosphodiesterase-5 inhibitors, while 59.0% reported no use of any erectile aids (n = 978). Some patients used multiple erectile aids, hence the proportions do not sum to 100%. Inclusion of erectile aid use improved the RMSE from 17 to 16 when using complete data through 12 months to predict 24-month outcomes but not at baseline.

Inclusion of all the new predictors (surgical volume, nerve sparing, and erectile aid use) in a final model did not significantly improve the 24-month EPIC-26 score outcome. With the addition of these variables, the model had similar performance at baseline (RMSE 26, MAE 22, AUC 0.81) and with complete information through 12 months (RMSE 17, MAE 12, AUC 0.93).

Discussion

We developed a dynamic model for predicting recovery of sexual function among patients undergoing RP. Our model exhibits good performance for preoperative predictions and even better performance for dynamic predictions, which facilitates the provision of realistic expectations for patients preoperatively and during the survivorship period. Although the dynamic component of the model may not impact patients’ decision-making for surgery, the dynamic component of the tool may potentially reduce decision regret and reassure patients who have experienced only partial recovery of sexual function in the early postoperative period. For example, using the baseline model, a patient with a preoperative EPIC-26 sexual domain score of 57 would be predicted to have a 24-mo domain score of 46 (Supplementary Table 2). However, using the dynamic model, a patient with the same characteristics and a score of 57 at 6 months after surgery would be expected to ultimately end up with a score of 81 on the basis of their expected recovery trajectory.

While most prediction models in prostate cancer have focused on survival-related outcomes (eg, margin status, disease free survival) [28], [29], patients undergoing RP do have a strong focus on sexual function, which should be carefully considered in the decision-making process. Patients put themselves at substantial risk in pursuit of cancer-free status; prediction of erectile function outcomes improves their ability to evaluate the tradeoffs involved in RP and gives clarity regarding realistic postoperative goals.

Our model improves on models previously developed by other investigators [13], [14]. A 2011 prediction model had AUC of 0.77 for predicting 24-month erection quality in an RP cohort (n = 524) [14]. In comparison, our model was developed on a fourfold larger cohort (n = 4115 at 12 months and n = 2313 at 24 months), achieves a higher AUC (0.80), and can provide updated predictions for 24 months at the 12-month mark with even more precision (AUC 0.88; Table 2). A more recent model predicted the 5-yr EPIC-26 sexual domain score with RMSE of 26 and MAE of 21 using data from the CEASAR registry [13]. Our study also predicts erection quality, which is notable given that this outcome may be more interpretable by patients. Addition of postoperative information improved the RMSE to 18 and MAE to 13 for the model, highlighting how the dynamic component of the model can tailor predictions further during the survivorship period. Predictors are similar between this model and previously described models with the exception of a novel predictor in our model, PROMIS Satisfaction with Sex Life, which we have previously described as an important component of sexuality for prostate cancer survivors [23]. To increase the utility of the model, we did not include nerve sparing in this model given that the extent of nerve sparing that will be performed is unknown at baseline (ie, before surgery) and nerve sparing varies by individual surgeon; our model focuses on patient-level determinants. In addition, sensitivity analyses demonstrated that inclusion of nerve sparing did not improve the model performance.

Our study does have limitations. We only evaluated practices in the state of Michigan, although the practices had diverse patient cohorts, sizes, and provider types, including both high- and low-volume surgeons. We did not include outcomes beyond 24 months. Although recovery of sexual function is uncommon beyond 24 months, there is some evidence that sexual activity recovery after surgery in some cases peaks at 3–6 yr after surgery [30], [31]. In addition, this model is only applicable to patients considering RP and does not predict sexual function following other prostate cancer treatments such as radiation therapy. We acknowledge that patients have a choice of more than one therapy (eg, RP, radiotherapy, active surveillance), all of which differ in their impact on HRQOL [32], [33], [34], [35], [36]. RP has the most deleterious effect on sexual function, an important component of HRQOL, and therefore patients considering RP would benefit the most from a prediction model for sexual function recovery [37]. Lastly, we did not include the impact of the surgeon in the final model. We examined the impact of surgeon volume and noted it did not improve the model performance beyond patient-level determinants. Thus, we focused on patient-oriented factors to make our work more generalizable. The model nonetheless has excellent performance and omission of this variable gives the model expanded utility (i.e., for use in practice settings outside of MUSIC).

Despite these limitations, our findings have significant implications for patients, providers, policymakers, and hospital leaders. The results have immediate relevance for patients facing therapy decisions for early-stage prostate cancer. For the first time, we have produced a dynamic model that allows personalized prediction of sexual function recovery before and after surgery. This may be most relevant for patients for whom preoperative assessment suggests that sexual function is their most important recovery domain. Providers can now, in an evidence-based way, support patients in making informed decisions and provide an individualized sexual function trajectory during survivorship. These results are important for policymakers and hospital leaders who are looking at ways to assess quality. Evidence-based precision tools for predicting outcomes can be used to improve outcomes and facilitate the provision of high-quality care. We have the unique opportunity to change the face of the delivery of prostate cancer care from decision-making to survivorship with precision medicine. While survival and oncologic outcomes are critical and at the forefront, patients who are cured of their prostate cancer with RP deserve an evidence-based way to understand the impact of surgery and treatments on their sexual function. A user-friendly prototype has been developed for utilization.

Conclusions

An evidence-based, personalized approach to understanding the impact of RP on sexual function and to guide decision-making is imperative. In this study we developed and validated a dynamic model to predict sexual function before and after RP using real-time inputs. If patients are provided with realistic expectations and a personalized approach to predicting sexual function before and after surgery, patients may have lower decision regret, better evidence-based decision-making, and a better understanding of their individual trajectory for sexual function recovery after RP.

: Nnenaya Agochukwu-Mmonu and Karandeep Singh had full access to all the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis.

Study concept and design: Agochukwu-Mmonu, Singh, Denton, Wittmann, Miller.

Acquisition of data: Agochukwu-Mmonu, Murali, Singh.

Analysis and interpretation of data: Agochukwu-Mmonu, Murali, Singh.

Drafting of the manuscript: Agochukwu-Mmonu, Singh.

Critical revision of the manuscript for important intellectual content: Agochukwu-Mmonu, Murali, Singh, Denton, Wittmann, Dunn, Montie, Peabody, Miller.

Statistical analysis: Murali, Singh.

Obtaining funding: Agochukwu-Mmonu, Montie, Miller.

Administrative, technical, or material support: Denton, Miller, Singh, Peabody.

Supervision: Agochukwu-Mmonu, Wittmann, Denton, Miller, Singh.

Other: None.

Nnenaya Agochukwu-Mmonu certifies that all conflicts of interest, including specific financial interests and relationships and affiliations relevant to the subject matter or materials discussed in the manuscript (eg, employment/affiliation, grants or funding, consultancies, honoraria, stock ownership or options, expert testimony, royalties, or patents filed, received, or pending), are the following: None.

This project was funded by a grant from University of Michigan Department of Precision Health. The Michigan Urological Surgery Improvement Collaborative is supported by Blue Cross Blue Shield of Michigan (BCBSM) as part of the Blue Cross and Blue Shield of Michigan Value Partnerships program. The opinions, beliefs, and viewpoints expressed herein do not necessarily reflect those of Blue Cross Blue Shield of Michigan or any of its employees.

The corresponding author would like to thank all the staff and Urologists from the Michigan Urological Surgery Improvement Collaborative.