A Systematic Review and Meta-Analysis of Prognostic Nomograms After UTUC Surgery.

Background: Current guidelines recommend assessing the prognosis in high-risk upper tract urothelial carcinoma patients (UTUC) after surgery. However, no specific method is endorsed. Among the various prognostic models, nomograms represent an easy and accurate tool to predict the individual probability for a specific event. Therefore, identifying the best-suited nomogram for each setting seems of great interest to the patient and provider.
Objectives: To identify, summarize and compare postoperative UTUC nomograms predicting oncologic outcomes. To estimate the overall performance of the nomograms and identify the most reliable predictors. To create a reference tool for postoperative UTUC nomograms, physicians can use in clinical practice. Design: A systematic review was conducted following the recommendations of Cochrane's Prognosis Methods Group. Medline and EMBASE databases were searched for studies published before December 2021. Nomograms were grouped according to outcome measurements, the purpose of use, and inclusion and exclusion criteria. Random-effects meta-analyses were performed to estimate nomogram group performance and predictor reliability. Reference tables summarizing the nomograms' important characteristics were created.
Results: The systematic review identified 26 nomograms. Only four were externally validated. Study heterogeneity was significant, and the overall Risk of Bias (RoB) was high. Nomogram groups predicting overall survival (OS), recurrence-free survival (RFS), and intravesical recurrence (IVR) had moderate discrimination accuracy (c-Index summary estimate with 95% confidence interval [95% CI] and prediction interval [PI] > 0.6). Nomogram groups predicting cancer-specific survival (CSS) had good discrimination accuracy (c-Index summary estimate with 95% CI and PI > 0.7). Advanced pathological tumor stage (≥ pT3) was the most reliable predictor of OS. Pathological tumor stage (≥ pT2), age, and lymphovascular invasion (LVI) were the most reliable predictors of CSS. LVI was the most reliable predictor of RFS. Conclusions: Despite a moderate to good discrimination accuracy, severe heterogeneity discourages the uninformed use of postoperative prognostic UTUC nomograms. For nomograms to become of value in a generalizable population, future research must invest in external validation and assessment of clinical utility. Meanwhile, this systematic review serves as a reference tool for physicians choosing nomograms based on individual needs. Systematic Review Registration: https://www.crd.york.ac.uk/prospero/display_record.php?RecordID=282596, identifier PROSPERO [CRD42021282596].

1 Introduction

Upper tract urothelial carcinoma (UTUC) is a rare and biologically heterogeneous disease that accounts for less than five percent of all urothelial tumors (1). Given the disease’s heterogeneity, risk stratification leads the patient’s management.

Preoperative risk assessment guides the selection of treatment strategies in patients with localized disease, recommending kidney sparing surgery for low-risk (2) and radical nephroureterectomy for high-risk patients (3–6). Postoperative risk stratification decides on the administration of adjuvant chemotherapy and defines the follow-up strategy (7). For this purpose, the European Association of Urology guideline recommends using prognostic models (7). However, no specific model has been endorsed yet.

Improvements in postoperative patient counseling regarding adjuvant treatment are urgently needed. The POUT trial and Checkmate-274 provided evidence for a disease-free-survival benefit of adjuvant therapy (platin-based chemotherapy, nivolumab) (8–10). Conversely, IMvigor010, evaluating adjuvant immunotherapy with atezolizumab, failed to demonstrate any benefit (11). Patient risk stratification might explain these differences. The supplementary analysis of IMvigor010 showed that TNM-based risk stratification was insufficient in identifying patients in need of adjuvant treatment (12).

Among the various prognostic models, nomograms represent a user-friendly tool to predict a patient’s individual probability for a specific event, such as tumor recurrence or death (13–15). This information helps to individualize medical care and counsel patients based on evidence.

Over the past decades, various nomograms have been developed for postoperative UTUC patient counseling. However, there is no comprehensive overview to guide potential users regarding the utility or accuracy of these tools. It is, indeed, necessary to summarize the most reliable nomograms for clinical practice, identify those applicable for further research, and give suggestions for individual patient settings.

We performed a systematic review and meta-analyses on multivariable postoperative prognostic UTUC nomograms predicting oncological outcomes. Our secondary objectives were to outline and compare the nomograms, investigate their overall performance, identify the most reliable predictors, and provide physicians with a reference tool for clinical practice.

2 Material and Methods

We performed this review following the recommendations of the Cochrane Prognosis Methods Group (16). The review protocol was prospectively registered in PROSPERO (registration number: CRD42021282596).

2.1 Search Strategy

We used the CHARMS checklist for systematic reviews of prediction modeling studies and the PICOTS scheme to define the review question (17). We searched for all studies that included UTUC patients (P), where a multivariable prognostic nomogram was investigated (I), to predict the oncologic outcome (O) (overall survival [OS], cancer-specific survival [CSS], recurrence-free survival [RFS], or intravesical recurrence [IVR]) in a one, three, or five years period (T), and that can be used after surgery or at a specific time point along the further course of the disease (postoperative) (S). Surgery was defined as any surgery intended to remove the tumor entirely.

Studies were eligible if (I) they matched the research question and (II) presented data on the development and internal or external validation of a multivariable prognostic nomogram. The studies also had to present data on (III) the nomogram’s prediction accuracy and calibration. (IV) Only full-text manuscripts published in English were included.

We searched the electronic databases Medline and EMBASE for studies published before December 2021. The search string used is listed on the PROSPERO website.

2.2 Data Collection

2.2.1 Study Inclusion and Exclusion

Two reviewers independently screened the titles and abstracts to identify eligible publications and performed a full-text review based on the inclusion criteria. Disagreements between the two reviewers were resolved in consensus with the co-authors. shows the PRISMA flow chart (18).

Figure 1

This figure shows the PRISMA flow chart of the study selection process. From: Page et al. (18).

2.2.2 Data Extraction and Management

Two reviewers independently extracted the data to a predefined datasheet, and a third reviewer verified the accuracy of the extraction process. We extracted data from the following domains: overall information, paper information, source data, participant information, outcomes to be predicted, model development, model validation, sample size, predictors, model performance, internal validation, and external validation. lists the data extracted in detail.

2.2.3 Model Performance Measures to be Extracted

We assessed the performance of each nomogram by extracting the measures for discrimination (c-Index with 95% confidence intervals [CIs] and standard error [SE], area under the receiver operating curve) and calibration (calibration plot interpretation, observed/expected ratio) presented without validation, and on internal and external validation. We assessed the independent effect of each predictor by extracting the hazard ratio (HR) or the coefficient Beta with 95% CIs and SE presented in the final model.

2.2.4 Dealing With Missing Data

We calculated missing data of performance measures as recommended by Cochrane (16). The predictor’s Beta and Beta’s and HR’s SE were calculated with the given HR and the 95% CIs. Missing 95% CIs of HRs were calculated either with the SE or with the HR and its p-value (19). Missing 95% CIs and SE of the c-Index were calculated with the c-Index and the number of patients with and without events (20). All the corresponding authors were contacted in case of missing data.

2.3 Quality Assessment

2.3.1 Assessment of Risk of Bias

Two reviewers independently assessed the risk of bias (RoB) of the included studies, using the dedicated Prediction model Risk Of Bias ASessment Tool (PROBAST) (21), which considered four potential sources of bias and three of applicability. The results of the PROBAST analysis were reported for each domain (bias: low risk, high risk, unclear; applicability: low concern, high concern, unclear), and an overall score for RoB and applicability was given.

2.3.2 Assessment of Nomogram Heterogeneity

To account for nomogram heterogeneity, we stratified them into groups (A, B, C, D, E) regarding the purpose of use (after surgery or at the time of intravesical recurrence) and the outcomes to be predicted (OS or CSS or RFS or IVR). Group A included nomograms predicting OS, group B: CSS, group C: RFS, and group D: IVR after surgery. Group E included nomograms predicting CSS at the time of intravesical recurrence. We further stratified the nomograms from group B according to the surgical treatment (all types of surgery or radical nephroureterectomy only) and the patients’ baseline inclusion and exclusion criteria (neoadjuvant systemic treatment, adjuvant systemic treatment). Group B1 included nomograms for all types of surgery, whereas groups B2 and B3 included radical nephroureterectomy nomograms without systemic neoadjuvant treatment. Further, group B2 included nomograms with adjuvant and B3 without adjuvant systemic treatment. The nomogram groups were used for further statistical analysis. highlights the nomogram group stratification process and lists the nomograms within each group.

Figure 2

This figure shows the nomogram group stratification process. Further, it lists all nomograms within each group.

2.4 Data Synthesis

2.4.1 Summary of Nomograms

We summarized the key findings of the included studies by giving (I) general information on the publication, (II) the nomogram’s purpose of use, (III) predicted outcomes, (IV) validation types, (V) inclusion and exclusion criteria of the study, (VI) essential patient characteristics, (VII) nomogram predictors, (VIII) nomogram performance (without validation, on internal validation, on external validation), (IX) and the RoB and applicability of the publication/nomogram.

2.4.2 Meta-Analysis Approach

We conducted the meta-analyses by nomogram groups due to the lack of validation studies and nomogram heterogeneity.

We investigated the overall performance of the nomogram groups by pooling the c-Index, which is a measure of discrimination accuracy. It accounts for censored data and is frequently used with survival data. Its value ranges between zero and one, with a value of 0.5 indicating prediction by chance (22). We set the cut-off values 0.6, 0.7, and 0.8 for moderate, good, and excellent nomogram discrimination accuracy. We included the c-Index that accounted best for the risk of overfitting in development studies: 1. Validation with an internal split cohort, 2. Validation with resampling by bootstrapping or cross-validation, 3. Development cohort without validation. We included the c-Index of all external validation studies.

We identified the most reliable predictors within each nomogram group by pooling Beta for predictors with a similar definition. The coefficient Beta is a measure of the predictor’s effect, and its value is independent of the measurement scale and therefore comparable among different variables. Positive values indicate a determinate effect, whereas negative values indicate a beneficial effect (23).

The number of three measurements (c-Index, Beta) was set as the lower limit for pooling. Therefore, we did not pool Beta within nomogram groups D and E and the c-Index for nomogram group E.

We used a Frequentist approach random-effects meta-analysis with the restricted maximum likelihood estimation and the Hartung-Knapp correction for calculating confidence intervals. If less than five studies were included, we additionally conducted a Bayesian approach random-effects meta-analysis. The meta-analyses results were plotted on a forest plot. The significance of the pooled summary estimate was assessed with the 95% CI, and the prediction interval (PI) verified its consistency. We used the statistical software R (v.4.0.5/2021) using the packages’ meta’ (24), ‘metafor’ (25), and ‘metamisc’ (20).

3 Results

3.1 Nomogram Search and Study Characteristics

From the 1524 records identified, we performed a full-text review of 282 articles and finally included 21 studies (26–46) for the systematic review and 17 studies (26–30, 32–35, 38–41, 43–46) for the meta-analyses. The full-text review excluded four studies that presented prognostic nomograms for UTUC patients receiving various treatments (surgery and/or radiotherapy and/or chemotherapy) (47–50). Nineteen studies presented nomogram development and internal validation data (26–35, 37, 39–46), of which two additionally presented external validation data of a separate nomogram (37, 44). Two studies presented external nomogram validation data only (36, 38). The development cohorts included 21,610 patients, and the internal split and external validation cohorts included 14167 patients. Patient data were collected in Asia in 11 (28–30, 33, 34, 36–38, 41, 44, 46), in North America in seven (26, 30–32, 41, 42, 45), in Europe in three (40, 42, 43), and worldwide in three (27, 35, 39) studies.

We identified 26 postoperative prognostic nomograms. All nomograms had been developed based on a cox-regression model. gives a detailed overview of the studies and nomograms, taking study inclusion and exclusion criteria into consideration. lists the patient characteristics of development and validation studies in detail.

Table 1

This table summarizes the publications included in the systematic review, highlighting nomogram prediction outcome, nomogram validation, and patient inclusion and exclusion criteria.

GENERAL INFORMATION			ENDPOINTS								VALIDATION				INCLUSION / EXCLUSION CRITERIA											COMMENTS
First author	Year	Purpose (FS = following surgery, IV = intravesical recurrence, V = validation)	Overall Survival	OS timeframe (years)	Cancer Specific Surival	CSS timeframe (years)	Recurrence Free Survival	RFS timeframe (years)	Intravesical Recurrence	IVR timeframe (months)	Internal Validation Bootstrapp-/Resampling	Internal Validation Split Cohort / 2nd Cohort	External Validation Paper	External Validation Paper Author / Year	I/E Radical Nephroureterectomy	I/E Surgery other than RNU	I/E Systemic Neoadjuvant Treatment	I/E Systemic Adjuvant Treatment	I/E Intravesical Adjuvant Treatment	I/E Systemtic Palliative Treatment	I/E Badder Cancer~	I/E Contralateral UTUC~	I/E Other Malignancy or Systemtic Disaese	other important I/E	Nomogram groups	Y = yesN = noNI = no information~if specifically named / otherwise other malignancy
Abdul-Muhsin H	2021	FS	Y	5	Y	5	Y	5	N		N	Y	N		I	E	I	I	NI	NI	NI	NI	NI	E: no currative intent	A, B1, C
Cha EK	2012	FS	N		Y	2; 5	Y	2; 5	N		N	Y	Y	Zeng S 2019	I	E	E	E	NI	NI	E*	NI	NI		B1, B3, C	*no previous MIBC
Chen J	2021	FS	Y	3; 5	N		N		N		N	Y	N		I	E	E	I	NI	NI	NI	NI	E		A
Chen X	2020	FS	Y	3; 5	N		N		N		Y	N	N		I	E	NI	NI	NI	NI	I	NI	E		A
Fang D	2020	FS	Y	3; 5; 10	N		N		N		N	Y	N		I	I	E	E	NI	NI	NI	E	NI	E: solitary kidney	A
Hou G	2020	IV	N		Y	1; 3; 5	N		N		Y	N	N		I	E	NI	NI	NI	NI	E*	E	NI	E: systemic recurrence before intravesical recurrence; more than one intravesical recurrence	E	*no bladder cancer before RNU
Jeldres C	2010	FS	N		Y	5	N		N		N	Y	N		I	E	NI	NI	NI	NI	NI	NI	NI		B1, B2
Kim M	2015	FS	N		Y	2; 5	Y	2; 5	N		Y	N	N		I	E	E	I	NI	NI	E*	NI	E		B1, B2, C	*no previous MIBC
Kim S	2019	FS	N		N		Y	3	N		Y	Y	N		I	E	E	I	NI	NI	I	E	NI	E: previous or concurrent radical cystectomy	C
Krabbe LM	2017	FS	N		N		Y	5	N		Y	Y	N		I	I	NI	NI	NI	NI	I	NI	NI	E: low grade UTUC	C
Ku J	2013	V													I	E	E	I	NI	NI	NI	NI	NI	E: previous or concurrent cystectomy
Lai S 1	2021	V													I	E	E	I	NI	NI	E*	E	NI			*no previous or conocmittant muscle invasive bladder cancer
Lai S 2	2021	FS	N		N		N		Y	12; 36; 60	Y	Y	N		I	E	NI	I	I	NI	E*	E	NI		D	*no synchronous baldder cancer, previous bladder cancer possible
Roupret M	2013	FS	N		Y	5	N		N		Y	Y	N		I	E	E	E	NI	NI	E*	NI	NI	E: pT0	B1, B3	*exclusion history of muscle invasive bladder cancer
Seisen T	2014	FS	N		Y	5	N		N		Y	Y	N		I	E	E	NI	NI	NI	E*	NI	NI	E: pTa, pT4, N 1-2, M1	B1, B2	*exclusion history of muscle invasive bladder cancer
Wang M	2021	FS	Y	1; 3; 5	N		N		N		N	Y	N		I	I	I	I	I	I	E	NI	NI	Nomogram I: HG only, Nomogram II: LG only	A
Xylinas E	2014	FS	N		N		N		Y	3; 6; 12; 18; 24; 36	N	Y	Y	Lai S 1 2021 Lai S 2 2021	I	E	E	I	NI	NI	E*	NI	NI		D	*no previous muscle invasive bladder cancer and high grade non muscle invasive bladder cancer
Yates DR	2012	FS	N		Y	3; 5	N		N		Y	Y	Y	Ku J 2013	I	E	NI	NI	NI	NI	NI	NI	NI		B1, B2
Zeng S	2019	FS	N		Y	3; 5	N		N		Y	Y	N		I	E	E	E	NI	NI	I	NI	NI		B1, B3
Zhang G	2019	FS	Y	3; 5	Y	3; 5	N		N		N	Y	N		I	I	NI	NI	NI	NI	NI	NI	E		A, B1
Zhang X	2020	FS	N		N		N		Y	24; 48	N	Y	N		I	E	E	I	I	NI	I	E	NI	E: pTa, pT4, N1-2, M1	D

3.2 Nomogram Predictors

The median number of predictors used in the nomograms was 5.5 (Range 2-9). The most frequent predictors were pathological T-stage (n=21), age (n=17), pathological N-stage (n=16), tumor grade (n=12), and lymphovascular invasion (LVI) (n=11). In four nomograms, the reported HRs of the multivariable analysis and the assigned weights of the nomogram predictors did not match (42–44). gives the predictors of each nomogram in detail. summarizes the predictors most frequently used within each nomogram group.

3.3 Nomogram Performance Measures

The c-Index of development studies ranged from 0.657 (95% CI 0.560-0.755) (34) to 0.825 (95% CI 0.648-1) (41). Calibration plots of development studies showed moderate to good nomogram calibration, except for one nomogram (43).

The c-Index of external validation studies ranged from 0.683 (38, 42) to 0.742 (36, 43). A calibration plot of external validation was presented for only three nomograms (36–38, 42, 43). Nomogram calibration was good for two nomograms (37, 38, 42) but poor for the other (36, 43).

Neither development studies nor validation studies reported the observed/expected ratio. outlines the performance measures reported for the nomograms.

Table 2

This table gives a detailed overview of the performance of the nomograms (discrimination = c-Index, and calibration = interpretation of the calibration plot) on development and validation studies.

INFORMATION				Abdul-Muhsin H						Cha EK				Chen J		Chen X	Fang D	Hou G	Jeldres C	Kim M		Kim S
				2021						2012				2021		2020	2020	2020	2010	2015		2019
OUTCOME / GROUP				Overall survival		Cancer Specific Survival		Metasatasis Free Survival		Cancer Specific Surival		Recurrence Free Survival		Overall Survival		Overall Survival	Overall Survival	Cancer Specific Survival	Cancer Specific Survival	Cancer Specific Survival	Recurrence Free Survival	Recurrence Free Survival
				A		B1		C		B1, B3		C		A		A	A	E	B1, B2	B1, B2	C	C
RESULTS	c-Index Development Paper¹			0.784		0.714		0.753		0.815		0.768		0,804 (95%CI 0,713-0,895)		0.82	0,698 0,724	NI	0,753	0,802 (95%CI 0,752-0,851)	0,788 (95%CI 0,73-0,826)	0,657 (95%CI 0,560-0,755)
	c-Index External Validation			NI		NI		NI		0,69 0,7		NI		NI		NI	NI	NI	NI	NI	NI	NI
	Calibration Plot Development Paper Interpretation Authors 3 years / 5 years (+/~/-)1,2			NI / +		NI / +		NI / +		NI / +		NI / +		NI / +		+ / +	~ / NI	+ / +	NI / +	NI / +	NI / +	+ / NI
	Calibration Plot External Validation Interpretation Authors 3 years / 5 years (+/~/-)²			NI		NI		NI		NI / NI		NI		NI		NI	NI	NI	NI	NI	NI	NI
INFORMATION				Krabbe LM		Lai S 2		Roupret M		Seisen T		Wang M				Xylinas E		Yates DR	Zeng S	Zhang G		Zhang X
				2017		2021		2013		2014		2021				2014		2012	2019	2019		2020
OUTCOME / GROUP				Recurrence Free Survival		Intravesical Recurrence		Cancer Specific Survival		Cancer Specific Survival		HG Group Overall Survival		LG Group Overall Survival		Intravesical Recurrence Full Model	Intravesical Recurrence Reduced Modell	Cancer Specific Survival	Cancer Specific Survival	Overall Survival	Cancer Specific Survival	No Intravesical Recurrence
				C		D		B1, B3		B1, B2		A		A		D	D	B1, B2	B1, B3	A	B1	D
RESULTS	c-Index Development Paper¹			0,76 (±0,012)		NI		0,79 (95% CI 0,75-0,83)		0,8 (95%CI 0,76-0,84)		0,729 (95%CI 0,707-0,750) 0,763 (95% CI 0,656-0,869)		0,731 (95%CI 0,67-0,791) 0,825 (95%CI 0,648-1)		0.69	0.678	0.78	0,73 (95%CI 0,59-0,87)	0,702 (95%CI 0,684-0,720)	0,771 (95%CI 0,746-0,796)	0,678 (95%CI 0,583-0,772)
	c-Index External Validation			NI		NI		NI		NI		NI		NI		0.684	0.683	0.742	NI	NI	NI	NI
	Calibration Plot Development Paper Interpretation Authors 3 years / 5 years (+/~/-)1,2			NI / ~		+ / NI + / NI		NI / ~		NI / +		+ / +		+ / +		+ / NI	+ / NI	+ / -	~ / ~	+ / +	+ / +	+ / NI
	Calibration Plot External Validation Interpretation Authors 3 years / 5 years (+/~/-)²			NI		NI		NI		NI		NI		NI		+ / NI + / NI	+ / NI	- / -	NI	NI	NI	NI

¹We only report the c-Index/calibration plot accounting best for overfitting (split cohort validation > bootstrapping/resample validation > development cohort).

²If calibration plot was given but without interpretation from the authors, the reviewers interpreted the calibration plot. NI, no information given.

3.4 Nomogram Reference Tool

, , , and , represent a reference tool for postoperative UTUC nomograms. As a first step, shall be used to identify nomograms predicting the outcome of interest. As a second step, needs to be checked for inclusion and exclusion criteria to be considered. If more than one nomogram is applicable, can be used to compare the nomogram’s diagnostic accuracy and calibration presented on internal and external validation cohorts. As the last step, , can be checked to evaluate whether the patient’s characteristics align with the nomogram’s development and validation cohort and whether the predictors are readily available. Using this stepwise approach, physicians can choose a nomogram that fits the individual patient’s needs.

3.5 Risk of Bias Assessment of Included Studies

For development studies, overall RoB was high in 100% of the studies. RoBs mainly were due to inconsistencies in the analysis (100%) and participants domains (54%). Most predictors were selected based on the results of a univariable analysis, and the complexity of the data was not considered. Moreover, the data source and patient inclusion and exclusion criteria had a high RoB. Overall applicability of development studies was unclear in 42% of the studies, mainly due to inconsistencies in the participants (31%) and predictors (11%) domains.

For validation studies, overall RoB was high in 100% of the studies. RoBs mainly were due to inconsistencies in the analysis (60%) and participants (60%) domains. All studies validated the nomograms with retrospective cohorts, and in most cases, this resulted in a high RoB. Furthermore, most validation studies did not report handling of missing data and did not update the nomograms. Overall, the applicability of the validation studies was good.

summarizes the PROBAST assessment of nomogram development and validation studies. reports the PROBAST assessment of all nomograms.

Figure 3

PROBAST summary (RoB domains and applicability domains) for all nomogram development (A, B) and validation studies (C, D) included in this systematic review.

3.6 Meta-Analyses

3.6.1 Pooled Predictor Coefficient Beta

Advanced pathological T-stage (≥ pT3) was a significant and consistent negative predictor (Beta summary estimate with 95%CI and PI > 0) of OS (Nomogram group A). Age, pathological T-stage (≥ pT2), and LVI were significant negative predictors (Beta summary estimate with 95%CI and PI > 0) of CSS (Nomogram group B1). LVI was a significant and consistent negative predictor (Beta summary estimate with 95%CI and PI > 0) of RFS (Nomogram group C). CSS subgroups B2 and B3 had no significant and consistent predictors. The maximum number of coefficients pooled per predictor was six. See – .

3.6.2 Pooled c-Index

Nomograms predicting OS (Nomogram group A), RFS (Nomogram group C), and IVR (Nomogram group D) had a significant and consistent moderate discrimination accuracy (c-Index with 95%CI and PI > 0.6). Nomograms predicting CSS (Nomogram group B1) had a significant and consistent good discrimination accuracy (c-Index with 95%CI and PI >0.7). CSS subgroup B2, but not B3, had a significant and consistent moderate discrimination accuracy (c-Index with 95%CI and PI >0.6). The maximum number of c-Indexes pooled per nomogram group was nine. See .

Figure 4

Summary forest plot of c-Index meta-analyses: The forest plot lists the results of individual meta-analyses. For each meta-analysis, the nomogram group, the statistical approach, the number of values included (n), the prediction interval (lower limit – upper limit), and the c-Index summary estimate (estimate and 95% CIs) are given.

4 Discussion

The discrepancy between nomogram development and external validation studies was high. Out of 26 nomograms, only four had been externally validated with a maximum of two validation cohorts. Indeed, the lack of external validation is a significant drawback for generalizability and discourages the uninformed use of nomograms in clinical practice. A common problem seen for prognostic model studies, as their five-year validation rate was shown to be only 16% (51).

We found that patients’ baseline characteristics varied widely between the studies. Patients had been recruited across different periods and continents and received varying treatment regimens. The increasing incidence of primary metastatic disease might have influenced nomograms’ predictions (52, 53). However, oncologic outcomes remained unchanged throughout the last decades (52). Further, the extent of surgical treatment can impact oncologic outcomes, either by increasing the risk of incomplete tumor resection or by selecting patients with favorable pathology (2). This is of particular importance, as previous studies highlighted the inaccuracy of preoperative staging (54). Even patients with relatively small tumors are at risk of muscle-invasive or even non–organ-confined disease (55). Similarly, excluding perioperative chemotherapy might select patients with favorable pathology or those unfit to receive the treatment. More importantly, it directly impacts the outcome (56). Heterogeneity in patient characteristics and its impact on oncologic outcomes is also a major drawback for the generalizability of these nomograms.

There were critical methodological weaknesses in the nomogram development studies. The data source mainly was from retrospective cohorts, which bears the risk of selection bias. Moreover, all development studies had a high RoB in the analysis domain. The nomograms might have been fitted to the characteristics of the development cohorts instead of a generalizable patient collective. In addition, no nomogram considered competing risks, which can cause an overestimation of the true event rate (57). Unfortunately, these limitations affect the reliability of the nomograms’ outcomes.

The meta-analyses identified several reliable nomogram predictors. Advanced pathological T-stage (≥pT3) predicted OS. Pathological T-stage (≥pT2), age, and LVI predicted CSS. LVI predicted RFS. These predictors were already known as individual risk factors (3, 58, 59). However, we elaborated that they retain their validity when combined, such as in a nomogram. Integrating novel biomarkers, reflecting the biological and clinical behavior of the tumor’s environment, could further improve the nomograms’ accuracy. So far, they have barely been considered.

The meta-analyses identified moderate discrimination accuracy for nomograms predicting OS, RFS, and IVR. Further, it identified good discrimination accuracy for nomograms predicting CSS. Because it was impossible to pool the c-Index of each nomogram separately, we could not identify the most accurate nomogram to be used in clinical practice. Instead, our analyses demonstrated the overall potential of postoperative prognostic UTUC nomograms, which justifies the effort for further research.

This systematic review highlights the critical absence of external validation studies, limiting nomograms’ applicability and uninformed use. Furthermore, it outlines that information on the clinical utility is scarce. Whether patients benefit from using nomograms remains unreported. However, improvements in postoperative risk stratification are urgently needed. Although the POUT trial and, most recently, Checkmate-274 demonstrated improved disease-free survival with adjuvant systemic therapy (platinum-based chemotherapy; nivolumab) in high-risk patients (10, 56), IMvigor010 failed to demonstrate any benefit (atezolizumab) (11). Further, the study raised concerns about postoperative TNM-based risk stratification (12). Therefore, future studies should focus on assessing the nomograms’ clinical utility and whether they can identify patients most suitable for adjuvant treatment.

This systematic review outlined similarities and differences between postoperative prognostic UTUC nomograms. Further, it provides physicians with a reference tool, enabling them to choose nomograms based on their individual needs and easily implement nomograms into clinical practice. For example, when searching for a nomogram predicting the five-year CSS following RNU, physicians can decide between the nomograms of Cha EK et al. (27) and Yates DR. et al. (43), as both have been externally validated. Further, they can decide whether to choose a nomogram taking the effects of adjuvant chemotherapy into account. As the last step, they can check whether the patient’s characteristics align with the patient cohort used for nomogram development or validation and whether the predictors are readily available. The reference tool will promote the widespread use of nomograms in postoperative UTUC patient counseling.

This study is the first systematic review summarizing postoperative prognostic UTUC nomograms. We set a standard for study quality, excluding all without internal validation data nor presenting discrimination and calibration accuracy. Although we used this most rigorous method, the approach could have missed potential nomograms. We estimated the nomograms’ and predictors’ overall predictive value by summarizing the c-Index and the coefficient Beta. Nevertheless, we could not estimate the overall nomogram calibration because the observed/expected rates were missing. However, calibration is essential to assess the benefit for clinical practice (60). We accounted for heterogeneity within the meta-analyses by stratifying nomograms into distinct groups. As a result, the number of studies included per analysis was low. Moreover, despite contacting all the corresponding authors in case of lacking results descriptions, our analyses were limited by missing values.

5 Conclusions

Despite a moderate to good discrimination accuracy, severe heterogeneity discourages the uninformed use of postoperative prognostic UTUC nomograms. For nomograms to become of value in a generalizable population, future research must invest in external validation and assessment of clinical utility. Meanwhile, this systematic review serves as a reference tool for physicians choosing nomograms based on individual needs.

Data Availability Statement

The original contributions presented in the study are included in the article/ . Further inquiries can be directed to the corresponding author.

Author Contributions

MP contributed to protocol/project development/management, data collection and management, data analysis and interpretation, and manuscript writing/editing. FK contributed to data collection and management, data interpretation, and manuscript writing/editing. AA, DD, EL, EX, FS, FQ, HM, LL, MRi, MRo, NS, PIK, PR, RSM, TK, TY, VM, and YL contributed to data interpretation and manuscript writing/editing. SFS contributed to protocol/project development/management, data interpretation, and manuscript writing/editing. BP contributed to protocol/project development/management, data collection and management, data interpretation, and manuscript writing/editing. All authors contributed to the article and approved the submitted version.

funding

The author PR is supported by a EUSP Scholarship from the European Association of Urology.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The reviewer DE declared a shared affiliation with the author EL to the handling editor at the time of review.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.