External Validation of the 2003 Leibovich Prognostic Score in Patients Randomly Assigned to SORCE, an International Phase III Trial of Adjuvant Sorafenib in Renal Cell Cancer.

PURPOSE: The 2003 Leibovich score guides prognostication and selection to adjuvant clinical trials for patients with locally advanced renal cell carcinoma (RCC) after nephrectomy. We provide a robust external validation of the 2003 Leibovich score using contemporary data from SORCE, an international, randomized trial of sorafenib after excision of primary RCC.
METHODS: Data used to derive the 2003 Leibovich score were compared with contemporary data from SORCE. Discrimination and calibration of the metastasis-free survival outcome were assessed in data from patients with clear-cell RCC, using Cox proportional hazards regression, Kaplan-Meier curves, and calculation of Harrell's c indexes. Secondary analyses involved three important SORCE groups: patients with any non-clear-cell subtype, papillary, and chromophobe carcinomas.
RESULTS: Four hundred seven recurrences occurred in 982 patients in the Leibovich cohort and 520 recurrences were recorded in 1,445 patients in the primary SORCE cohort. Clear discrimination between intermediate-risk and high-risk SORCE cohorts was shown; hazard ratio 2.74 (95% CI, 2.29 to 3.28), c-index 0.63 (95% CI, 0.61 to 0.65). A hazard ratio of 0.61 (95% CI, 0.53 to 0.70) confirmed poor calibration of the two cohorts. Discrimination was observed in secondary populations, with c-indexes of 0.64 (95% CI, 0.59 to 0.69) for non-clear-cell RCC, 0.63 (95% CI, 0.56 to 0.69) for papillary RCC, and 0.65 (95% CI, 0.55 to 0.76) for chromophobe RCC.
CONCLUSION: The 2003 Leibovich score discriminates between intermediate-risk and high-risk clear-cell and non-clear-cell RCC groups in contemporary data, supporting its use for risk stratification in adjuvant clinical trials. Over time, metastasis-free survival for patients with locally advanced RCC has improved. Contemporary data from adjuvant RCC trials should be used to improve prognostication for patients with RCC.

INTRODUCTION

The Leibovich score,[1] published in 2003, is widely used to guide postnephrectomy prognostication for patients with locally advanced renal cell carcinoma (RCC)[2] and for risk-stratifying patients into adjuvant clinical trials.[2]

CONTEXT

Key Objective

The 2003 Leibovich score guides the prognostication and the selection of clear-cell and non–clear-cell patients with locally advanced renal cell carcinoma (RCC) into clinical trials. Its up-to-date validation in contemporary data is necessary to support its continued use. To our knowledge, an evaluation of the 2003 Leibovich score's discrimination between risk groups for non–clear-cell RCCs has not previously been demonstrated.

Knowledge Generated

The 2003 Leibovich score demonstrated discriminative accuracy in contemporary clear-cell and non–clear-cell groups, supporting its use for recruiting and guiding the random assignment of participants to adjuvant RCC trials. Outcomes for patients with RCC have improved over time, rendering the 2003 Leibovich score poorly calibrated to contemporary outcomes.

Relevance

We support the use of the 2003 Leibovich score to risk-stratify patients with RCC suspected of being at intermediate or high risk of relapse.

Leibovich et al developed the score using retrospective data from patients with clear-cell RCC who underwent radical nephrectomy at the US Mayo Clinic between 1970 and 2000. Five features that were significantly associated with time-to-distant metastases (P < .001) comprised the final multivariable model: tumor category (6th TNM 2002), regional lymph-node status, maximum tumor diameter, nuclear grade, and presence of tumor necrosis. For clinical application, risk groups were defined as low (scores 0-2), intermediate (3-5), and high (6 or higher). Five-year metastasis-free probabilities were reported as 97.1%, 73.8%, and 31.2% respectively.[1]

The SORCE trial (ClinicalTrials.gov identifier: NCT00492258), evaluated the effect of sorafenib after nephrectomy and is one of the largest internationally recruiting randomized controlled trial in patients with locally advanced RCC, to date.[3] In SORCE, and now RAMPART (ClinicalTrials.gov identifier: NCT03288532),[4] the 2003 Leibovich score determines participant eligibility and guides their random allocation to trial arms.

Selection of the 2003 Leibovich score for this purpose is supported by its superior discriminative accuracy on direct comparison with several other prognostic scores.[5] Furthermore, the 2003 Leibovich score is simple to calculate. All score components are tumor-derived and routinely reported on RCC pathology, negating the need for additional expertise or training. Clinical markers such as patient's performance status are not included in the score, reducing the chance of subjective bias.

An external validation of the 2003 Leibovich score, using data from SORCE participants, was prespecified within the SORCE Protocol (online only). We focused on the intermediate-risk and high-risk patients as they are of specific interest for recruitment to adjuvant clinical trials. SORCE provided a large contemporary data set of individual participant data (IPD) with detailed and long follow-up. Unusually for a validation study, we accessed IPD used to derive the 2003 Leibovich score.[1] By creating closely matched data sets, we were able to compute measures of discrimination and calibration,[6-8] to directly compare the performance of the 2003 Leibovich score in the historical and contemporary cohorts. Accordingly, we provide a high-quality evaluation of the Leibovich score's ability to discriminate between patients at intermediate risk and high risk of relapse.

Although the 2003 Leibovich score is used in clinical trials that recruit patients with non–clear-cell RCC, its ability to stratify risk in this group has not been evaluated. Newer prognostic scores[9-11] have been developed (including some specifically for non–clear-cell subtypes), but none are commonly used in clinical trials, where straightforward application is key. We present the first exploration of the 2003 Leibovich score's discriminative accuracy within important histologic SORCE subpopulations: any non–clear-cell, papillary-only, and chromophobe-only carcinomas.

METHODS

Participants: Leibovich Score Calculation

SORCE participants were recruited from July 2007 to April 2013 from 147 centers in seven countries: United Kingdom, Australia, France, Belgium, the Netherlands, Spain, and Denmark, and followed up until July 2019.[3] Only patients with intermediate (3-5) or high (≥ 6) Leibovich scores were included in SORCE.[1,3] Participants with any histology except pure oncocytoma were eligible. Values for components of the 2003 Leibovich score (Data Supplement, online only) were prospectively collected for each participant on random assignment to SORCE.

The 6th TNM 2002 system was used by Leibovich et al and in SORCE. The same nuclear grading system that selects the worst WHO/International Society of Urological Pathology[12] features at each grade was used in both data sets (Data Supplement). In SORCE, this system pragmatically standardized grading across international trial sites and was used for all histologic subtypes including chromophobe and other non–clear-cell RCCs.

The SORCE trial was approved by national regulatory and ethical committees in each participating country and was conducted in accordance with the principles of Good Clinical Practice, the Declaration of Helsinki, and all applicable regulatory requirements and laws. All participants signed an informed consent form before entry into the study.

Participants and Outcomes: Leibovich and SORCE Populations

Two matched cohorts were analyzed. A derivation cohort derived from the 2003 Leibovich data set included patients with clear-cell RCC only and excluded the low-risk group (Leibovich scores 0-2). A validation cohort was derived from intermediate-risk and high-risk clear-cell RCC participants in SORCE. All patients in the 2003 Leibovich data set underwent radical nephrectomy. Partial or radical nephrectomy was permitted in SORCE, reflecting contemporary surgical practice.

The primary outcome was time to metastasis-free survival (MFS), defined as the interval between nephrectomy and the date of distant metastases. In the study by Leibovich et al,[1] deaths preceding presumed metastasis were treated as censored observations (C. Lohse, personal communication, April 2020). We defined MFS in the same way for this analysis. We censored time to MFS at 10 years in both cohorts to reflect available follow-up data in SORCE.

Secondary exploratory analyses were conducted in the three SORCE subpopulations: patients with any non–clear-cell histology, papillary-only, and chromophobe-only carcinomas.

Statistical Methods

Model validation was performed adhering to transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD) guidelines[13] (Data Supplement). The time origin used for both cohorts was date of surgery. A survival analysis allowing for late entry was used,[14] capturing the post hoc nonexposure of a SORCE participant to the risk of an MFS event between surgery and random assignment. In SORCE, 56/1711 (3%) dates of surgery were missing; they were estimated by taking a random selection of 56 values from the distribution of observed intervals between surgery and random assignment.

The performance of the 2003 Leibovich score was assessed using discrimination and calibration.[6,7] Discrimination denotes the ability of a model to distinguish between patients who have and have not experienced an event. Calibration relates to a model's predictive accuracy.

Discrimination was assessed graphically by observing the degree of separation between the Kaplan-Meier curves and by the hazard ratio (HR) between intermediate-risk and high-risk Leibovich risk groups in each cohort. We quantified discrimination according to Harrell's c-index,[8] which denotes the proportion of all usable patient pairs in whom the observed and predicted survival times are concordant. The c-index ranges from 0.5 (performance no better than chance) through to 1 (perfect discrimination).

Calibration measures agreement between predicted and observed outcomes. Good calibration is inferred if Kaplan-Meier curves for risk groups in the derivation and validation cohorts are similar. We quantified calibration through the HR of the indicator variable for the two cohorts (0 = derivation cohort, 1 = validation cohort) separately for the two risk groups (0 = intermediate risk, 1 = high risk). An HR around 1 suggests accurate calibration.

We also analyzed the ungrouped Leibovich scores 3,4,…,11, to compare the HRs between the individual scores and a base category (taken as score = 3). We fitted Cox models separately for the derivation and the validation data sets, with each individual score as the explanatory variable and graphed the results.

Furthermore, we analyzed the ungrouped scores as a single entity to compare the discrimination (c-index) of the Leibovich score with that of 2002 TNM staging.

The secondary (exploratory) analyses were conducted with the three SORCE subpopulations using same procedures as with the primary analysis.

All measures were reported with 95% CIs. P values were two-sided.

All analyses were performed in STATA (16.1; StataCorp LLC, College Station, TX).

RESULTS

The 2003 Leibovich data included 479 MFS events in 1671 US-based patients who had radical nephrectomies between 1970 and 2000. The SORCE data had 614 MFS events in 1711 patients enrolled between 2007 and 2013 (Fig 1). The derivation cohort included 407 MFS events in 982 patients with a median follow-up of 7.3 years (interquartile range, 3-10 years), whereas the validation cohort included 520 MFS events in 1,445 patients with a median follow-up of 7.2 years (interquartile range, 6.1-8.4 years; Fig 1).

FIG 1.

The primary analysis cohorts. MFS: time from nephrectomy to the date of distant metastases; deaths preceding metastasis were censored. aA nuclear-grade assignment was missing for one participant, which we imputed singly by substituting the most common nuclear grade value (3), to ensure completeness of the validation data set. MFS, metastasis-free survival.

Table 1 describes the demographic, clinical, and histologic characteristics of patients in the 2003 Leibovich data, the derivation cohort, the SORCE data set, and the validation cohort. The validation cohort included more high-risk than intermediate-risk patients (46% v 38%). The validation cohort included 652 (45%) patients who had a laparoscopic nephrectomy and 43 (3%) patients who had a partial nephrectomy, whereas all patients in the derivation cohort underwent radical open nephrectomy (Data Supplement). The median time to MFS in the derivation cohort was 9.2 years, whereas in SORCE, this was not reached within 10 years of follow-up.

TABLE 1.

Histopathologic Characteristics, Leibovich Score Components, and Median Follow-Up in Leibovich and SORCE Data

Primary Analysis Population: Discrimination and Calibration

Discrimination.

Figure 2 presents the results of the validation exercise graphically, showing Kaplan-Meier curves of MFS in the intermediate-risk and high-risk groups for each cohort. Figure 2 shows that discrimination between intermediate-risk and high-risk groups in the derivation cohort is substantial but not entirely maintained in the validation cohort. The c-index in the derivation cohort is 0.67 (95% CI, 0.65 to 0.69) compared with 0.63 (95% CI, 0.61 to 0.65) in the validation cohort (P = .01, chi square test).

FIG 2.

Kaplan-Meier curves for MFS by Leibovich risk group in the (A) derivation and (B) validation cohorts. In the validation cohort Kaplan-Meier plot, the number of patients entering at time 0 is given as 0 in the at-risk tables. It is a consequence of the late entry character of the follow-up data. Patients were not deemed at risk until they were randomly assigned into SORCE, which occurs after t = 0. MFS, metastasis-free survival.

Discrimination between high-risk and intermediate-risk groups in the derivation cohort, with intermediate-risk as the baseline category, is further indicated by an HR of 3.88 (95% CI, 3.18 to 4.74), compared with 2.74 (95% CI, 2.29 to 3.28) in the validation cohort. Thus, discrimination is maintained in the validation cohort, albeit significantly reduced (P = .003, interaction analysis) compared with the derivation cohort.

To assess whether the discrimination of the 2003 Leibovich score degrades over follow-up time, c-indexes at one and 10 years after nephrectomy were compared in both cohorts (Data Supplement). We show that although c-index values for the Leibovich score reduce over time, the difference is small in both data sets.

Calibration.

The validation and derivation survival curves for the intermediate-risk and high-risk groups are not aligned (Fig 2), suggesting poor calibration.

Overall, the MFS rate was 26% lower in the validation than in the derivation cohort (HR = 0.74; 95% CI, 0.65 to 0.85). For the intermediate-risk group, the reduction in MFS rate was 24% (HR = 0.76; 95% CI, 0.61 to 0.94), compared with 46% (HR = 0.54; 95% CI, 0.45 to 0.64) in the high-risk group. The results confirm a distinct lack of calibration between data sets.

Analysis of Ungrouped Leibovich Scores

Figure 3 shows that the HRs comparing individual scores with the reference category (Leibovich score 3) increase markedly as the score increases in both the derivation and validation cohorts, reflecting consistently higher discrimination with increasing Leibovich score. We combined groups with scores of 9 and above because very few patients had score 10 or 11, giving unreliable estimates. See the Data Supplement for Kaplan-Meier curves (Fig 1A), HR values (Data Supplement), and c-indexes for each score (Data Supplement). Lower values of c-index and HR for each score group in the validation cohort confirm in detail that discrimination is maintained, albeit attenuated, in the contemporary cohort.

FIG 3.

HRs estimated for ungrouped 2003 Leibovich scores in the derivation data set and in the validation data set. Values are presented with 95% CIs. The lowest score (3) in the validation data set is the reference category. HR, hazard ratio.

In both the validation and the derivation cohorts, the collapse of scores 3-5 and 6-11 into two larger prognostic groups (intermediate-risk and high-risk) results in reduced discrimination compared with the original Leibovich score. This is a compromise to achieve a clinically more useful risk stratification tool.

To compare the discrimination of the ungrouped 2003 Leibovich score with that of 2002 TNM, we calculated c-indexes using the primary analysis data sets. The Leibovich score outperformed the 2002 TNM system in the derivation cohort (c-indexes of 0.72 [SE 0.01] v 0.56 [SE 0.01]) and in the validation cohort (c-indexes of 0.67 [SE 0.01] v 0.56 [SE 0.01]).

Secondary Analyses: Discrimination

Three cohorts were included in the secondary analysis: those with any non–clear-cell RCC (N = 266; MFS events, n = 94), papillary RCC (N = 128; MFS events, n = 49), and chromophobe RCC (N = 96; MFS events, n = 21). Discrimination between intermediate-risk and high-risk groups within each SORCE subcohort was compared with that of the derivation cohort.

Figure 4 shows Kaplan-Meier estimates of MFS in the intermediate-risk and high-risk groups for SORCE non–clear-cell, papillary, and chromophobe populations. The maintained separation between the curves beyond 6 months indicates that the 2003 Leibovich score retains long-term discriminative capability in these SORCE subpopulations. Compared with the derivation cohort (c-index 0.67), we obtained c-indexes of 0.64 (95% CI, 0.59 to 0.69) for the SORCE non–clear-cell cohort, 0.63 (95% CI, 0.56 to 0.69) for SORCE papillary, and 0.65 (95% CI, 0.55 to 0.76) for the SORCE chromophobe group.

FIG 4.

Kaplan-Meier curves for MFS in (A) the derivation cohort and in the SORCE (B) non–clear-cell, (C) chromophobe, and (D) papillary subcohorts stratified by 2003 Leibovich risk group. Derivation cohort included for reference. In the validation cohort Kaplan-Meier plot, the number of patients entering at time 0 is given as 0 in the at-risk tables. It is a consequence of the late entry character of the follow-up data. Patients were not deemed at risk until they were randomly assigned into SORCE, which occurs after t = 0. MFS, metastasis-free survival.

An HR of 3.88 (95% CI, 3.18 to 4.74) between risk groups was observed in the derivation cohort, compared with 3.21 (CI, 2.05 to 5.03) for SORCE non–clear-cell patients, 2.61 (95% CI, 1.44 to 4.70) for papillary, and 3.88 (95% CI, 1.56 to 9.61) for the chromophobe cohort. Despite smaller cohort sizes with correspondingly larger imprecision, these results highlight the Leibovich score's preserved discrimination in these SORCE subpopulations.

Secondary Analyses: Calibration

Attenuated calibration between the SORCE subpopulations and the derivation cohort for each risk group is shown by observing the misalignment of the corresponding survival curves (Fig 4). This is quantified by HRs for MFS after fitting a Cox regression model to each risk group separately (Table 2). Five-year relapse probabilities (Table 3) show improved MFS compared with the corresponding derivation cohort in all SORCE subgroups. The difference is most marked between the high-risk groups.

TABLE 2.

HRs Comparing MFS for Each SORCE Subpopulation to the Derivation Cohort Separately in Intermediate-Risk and High-Risk Groups

TABLE 3.

Five-Year Survival Probabilities for Metastasis-Free Survival in the Derivation Cohort, the Validation Cohort, and Each SORCE Subcohort

DISCUSSION

Validation of the 2003 Leibovich score using contemporary IPD from a large international trial represents the highest quality of validation, according to the American Joint Committee on Cancer criteria for model selection.[15] We focused on the intermediate-risk and high-risk clear-cell patients, a group commonly recruited to adjuvant clinical trials. This study confirms that the grouped 2003 Leibovich score, although developed two decades ago, largely retains discrimination in the SORCE validation cohort (c-index 0.63; 95% CI, 0.61 to 0.65) when compared with the derivation cohort (c-index 0.67; 95% CI, 0.65 to 0.69). We therefore support its ongoing use for risk stratification in this setting.

Uniquely, we show that the 2003 Leibovich score discriminates comparably between intermediate-risk and high-risk patients in the non–clear-cell SORCE cohort (c-index 0.64; 95% CI, 0.59 to 0.69). Since the non–clear-cell cohort is limited by inherent variability in clinical trajectories, we explored the two largest non–clear-cell subtypes separately: papillary (c-index 0.63; 95% CI, 0.56 to 0.69) and chromophobe groups (c-index 0.65; 95% CI, 0.55 to 0.76). Although the latter analyses are limited by smaller patient numbers, they indicate negligibly attenuated discrimination compared with the derivation cohort.

Some of the immune-oncology–focused adjuvant RCC trials, including IMMOTION010 (ClinicalTrials.gov identifier: NCT03024996) and KEYNOTE-564,[16] use the TNM staging system for patient random assignment to trial arms. We show that discrimination of the 2003 Leibovich score exceeds that of 2002 TNM in the derivation cohort (c-indexes of 0.72 [SE 0.01] v 0.56 [SE 0.01]) and in the validation cohort (c-indexes of 0.67 [SE 0.01] v 0.56 [SE 0.01]). The improvement is noteworthy, considering that a c-index of 0.5 represents a performance that is no better than chance. This finding has implications for using TNM for participant selection to clinical trials. We also show that the 2003 Leibovich score loses discrimination over follow-up time, with c-indexes of 0.63 at 10 years compared with 0.69 at 1 year after surgery in SORCE (Data Supplement). We suggest that this small difference over long follow-up should not impact on the 2003 Leibovich score's use.

In 2018, Leibovich et al[10] published five scoring systems, modeling progression-free survival (PFS) and cancer-specific survival individually for clear-cell and papillary RCC and PFS for chromophobe carcinomas. A major tradeoff for histologic specificity is added complexity in terms of the number of scoring systems for different subtypes and models that comprise many more components for clear-cell RCC. This is important when considering trial practicalities including standardization and limiting the workload associated with assigning prognostic risk for eligibility purposes. In addition, the 2018 scores offers only minor improvement in discrimination for PFS and cancer-specific survival in clear-cell patients, with internally validated c-indexes of 0.83 and 0.86, respectively, versus 0.82 for MFS for the 2003 score.[1,10]

Overall, the simplicity, practical utility, and maintained discrimination in a multisubtype population shown by the 2003 Leibovich score support its standardized use for risk stratification in adjuvant RCC trials in preference to recently published, yet to be widely externally validated, subtype-specific scores.[9,10]

We were able to perform a robust calibration analysis using IPD from the original Leibovich study, matching risk groups and unifying the MFS definition across cohorts. We clearly demonstrate longer MFS in patients with intermediate-risk and high-risk clear-cell RCC in the validation cohort (5-year MFS; 78% [CI, 75 to 81] and 52% [CI, 48 to 56], respectively), compared with the corresponding derivation cohorts (5-year MFS; 72% [CI, 67 to 75] and 30% [CI, 25 to 35], respectively). Comparatively longer MFS for contemporary non–clear-cell, papillary, and chromophobe cohorts are also shown (Table 3). On the basis of this, it may be necessary to reconsider trial eligibility for patients with long-term low relapse risk, for example, those with intermediate-risk chromophobe RCC where 5 year MFS approaches 87% (CI, 75-94; Table 3). Overall, better outcomes for patients with locally advanced RCC over time corroborate findings in contemporary literature.[17,18] Improved MFS may be linked to factors such as improved radiologic and pathologic practices over time and the introduction of minimally invasive surgical techniques such as laparoscopic nephrectomy.[19] Differences may additionally reflect an evolution in renal tumor biology over time, driven by changing rates of modifiable risk factors such as obesity and smoking.

Our analysis is not without limitations. First, pathology samples were not centrally reviewed. However, strict guidance for their assessment was provided in the SORCE protocol. Second, patients with low Leibovich risk (score 0-2) were not included this validation, because they are usually cured by surgery or ablation and not usually considered for recruitment to adjuvant trials. We acknowledge that excluding the low-risk group is likely to have resulted in loss of some discrimination compared with that achieved by the complete Leibovich data. Third, our validation was performed using the whole SORCE cohort rather than being restricted to the placebo group. As SORCE showed a clear lack of benefit of sorafenib as an adjuvant strategy after nephrectomy, we considered that including patients from the experimental arms would have no detrimental impact on this analysis.

Finally, patient and tumor characteristics differed between the Leibovich and SORCE cohorts. The median age of SORCE patients was 5 years younger and included higher rates of T3a-4 tumors compared with the Leibovich cohort (67% v 52%). Other differences included higher rates of histologic tumor necrosis in the SORCE cohort (54% v 43%) and more nuclear grade 4 cases (18% v 10%) were present.

In time, it may be possible to improve upon outcome prediction in RCC by adapting prognostic scores to include immunologic or genetic biomarkers that show both prognostic and predictive benefit. An example is the transcript-based recurrence score,[20] which adds prognostic information when included with the 2003 Leibovich score. However, as it does not predict response to adjuvant treatment and is expensive and complex, it has not been routinely used.

Alongside prognostic and predictive biomarker studies, a pragmatic step will be to refine the 2003 Leibovich score by further unpicking the characteristics known to drive worse outcomes in RCC. A digital pathology review of SORCE tumor samples is underway. This will allow a comprehensive analysis of the heterogeneity among RCC tumor specimens.[21] It may also reveal further granularity within current 2003 Leibovich score features, to enhance the prognostication and the prediction of recurrence for patients with RCC. A practical goal will be to retain as much of the usefulness and simplicity of the original Leibovich score as possible.

In conclusion, the 2003 Leibovich score is a validated prognostic score which, in contemporary data, discriminates between patients with clear-cell RCC at intermediate risk and high risk of disease recurrence. In addition, it comparably discriminates relapse risk in patients with non–clear-cell, papillary, and chromophobe RCCs in our data set. Over time, MFS rates among patients have improved; therefore, clinicopathologic prognostic scores need to be regularly reviewed. With the wealth of data available from recent RCC trials, there is an opportunity to build upon the 2003 Leibovich score to better reflect the changing landscape of RCC.