Appraisal and use of a prognostic study from the urological literature.

Information about prognosis can be applied to research design and is essential for patient care and counseling. Prognostic study data is only useful if it is valid, transparent, and applicable to your patient. Using a clinical scenario and a relevant study from the urological literature, we outline a method to appraise a prognostic article, understand the results, and manage patients accordingly.

INTRODUCTION

Data from prognostic studies are used to define the natural history of disease, estimate prognosis, guide treatment decisions and counsel patients. While case-control studies and randomized controlled trials can provide prognostic information, cohort studies are best suited to identify and characterize associations between patient characteristics and outcome. The purpose of this article is to provide a framework for appraisal of a prognostic article in the urological literature, understand the results, and apply them to patient care.

CLINICAL CASE SCENARIO

A 46-year-old Caucasian man presents at your office with a solitary 8-cm left-sided renal mass detected on incidental imaging. Staging studies do not reveal evidence of lymphadenopathy or metastases. After reviewing management options, you perform a laparoscopic radical nephrectomy and hilar lymphadenectomy without incident. The pathological tumor characteristics were: pT2 N0 Mx clear cell renal cell carcinoma (RCC); 8 cm in greatest diameter; Fuhrman nuclear grade 3; no coagulative tumor necrosis; and negative surgical margins. The patient has a young family and is very concerned about his future. He asks you if he is likely to die from RCC. To accurately answer his question, you decide to review the evidence.

LITERATURE SEARCH

Based on the clinical scenario, you begin by formulating a concise clinical question: “In patients with clear cell RCC, what is the long-term survival following radical nephrectomy?” A MEDLINE search of the terms “renal cell carcinoma”, “nephrectomy” and “survival” identify a number of relevant articles. After browsing through the associated abstracts from, you select a seemingly appropriate article entitled “An outcome prediction model for patients with clear cell RCC treated with radical nephrectomy based on tumor stage, size, grade and necrosis: The SSIGN Score” by Frank et al.,[1] You decide to critically appraise your selected article using a framework specific to prognostic studies.[23]

STUDY SUMMARY

The Frank article presents data from a Mayo Clinic historical cohort study. They reviewed 1801 adult patients with unilateral clear cell RCC treated with radical nephrectomy between 1970 and 1998 at a single institution. Clinicopathological characteristics included age, gender, smoking history, signs and symptoms at presentation, 1997 TMN stage, tumor size, nuclear grade, histological tumor necrosis, sarcomatoid differentiation, cystic architecture, multifocality and surgical margin status. The primary outcome was cancer specific death. Mean follow-up was 9.7 years (range 0.1-31) with estimated cancer specific survival of 86.6%, 74.0%, 68.7%, 63.8% and 60.0% at 1, 3, 5, 7 and 10 years, respectively. Analysis revealed cancer-specific survival was independently associated with stage, tumor size, nuclear grade and coagulative tumor necrosis. Based on these associations, the authors generated a scoring system (SSIGN Score) to estimate prognosis.

When reviewing the article you ask key fundamental questions: Are the results valid? What are the results? Can the results be applied to my patient?

Are the results of the study valid?

You begin the critical appraisal process by determining whether effort was made to minimize bias and improve validity. To do so, we must ask four additional questions.

Was the patient sample representative?

It is important to keep in mind that a study group may not be directly representative of the general population or your individual patient. Exclusion criteria or referral patterns commonly result in systematic differences from the general population.[2] For example, a prognostic study carried out in a single tertiary care center may yield different results from those that contain all patients with disease within a defined geographical area. Such biases can alter outcomes if studied patients are more motivated, concerned about their health or from a higher socioeconomic stratum.[4]

Ideally, study authors will clearly define inclusion/exclusion criteria, how the disease was diagnosed, and demographic and disease specific factors.[5] In the Frank study, patients were excluded if they had non-clear cell histology, bilateral synchronous tumors, familial von Hippel-Lindau or tuberous sclerosis, Wilm's tumor or those less than 18 years of age. Furthermore, since all patients were treated at the Mayo Clinic, a large tertiary care hospital, the study sample may not represent the population at large. Since the publication by Frank et al., in 2002, there have been three external validations conducted by groups in Italy, Japan and Austria in 2006, 2008 and 2010.[6-8] All three found the Mayo Clinic SSIGN score was highly accurate in predicting outcome for their patients with RCC. Thus, you feel confident applying Mayo Clinic data to your patient population.

Were the patients homogeneous with respect to their prognostic risk?

For your patient, it would be ideal to find a study that consisted of healthy, 40-50 year old men with similar tumor characteristics. Realistically, these studies do not exist since they are not feasible and the results applicable to a small number of patients. Studies typically include a heterogeneous group to gain information on a wider range of patients. The study by Frank et al. included patients at varied disease stages and addressed heterogeneity by assessing associations between clinicopathological factors and prognosis.

Was follow-up sufficiently long and complete?

Many outcomes important to patients, such as cancer recurrence or death, may take a protracted period of time to occur. Therefore, to accurately estimate prognosis sufficiently long follow-up is necessary. Indeed, despite fulfilling all validity criteria, a study may be of limited use if the follow-up time was inadequate. In the Frank study, the median follow-up time for 649 living patients was 8.2 years. Therefore, a large number of patients had been followed for sufficiently long to allow survival estimates.

Equally important to duration is the completeness of follow-up. Prognostic study validity may be compromised if a significant number of patients are lost to follow-up. The reason is that those lost to follow-up may be systematically different from those who continue follow-up. In other words, loss to follow-up may bias the data and result in over or under estimates of prognosis.[2] Frank et al., did not report the number of patients lost to follow-up and the reasons why. However, they did report that 1,152/1801(64%) of patients were followed until they died and that 649/1801 (36%) were followed for greater than 8 years. Based on these data, we infer that a large majority of patients were followed for a prolonged period of time and do not believe loss to follow-up compromised the validity of the findings.

Were objective and unbiased outcome criteria used?

Outcome events in a prognostic study may be objective and easy to measure (death), require some judgment (death due to RCC), or require significant judgment (disease recurrence).[25] In the study by Frank et al., the outcome was RCC-specific death. This information was obtained from death certificates and while, there is some subjectivity in defining cause of death, this outcome is likely reliable.

To summarize, while the study sample is not population-based, the sample is likely representative of patients treated surgically in your practice. Included patients were at varied disease stages but this heterogeneity was addressed appropriately by stratifying and presenting results of similar patients based on common risk. There was adequate duration and completeness of follow-up and the primary outcome was objective. Thus, you deem the study to be of sufficient quality to produce valid results.

What are the results??

Since you are satisfied the results are valid you review the findings in detail by addressing two additional questions.

How likely are the results to occur over time?

Prognostic studies typically present results as the number of events that occur over time.[2] These can be described as a time period survival, which is the proportion of the cohort that experience an outcome in a defined period of time (e.g., 5-year survival) or median survival which is the time when half of the cohort has experienced the outcome. In the Mayo Clinic study, estimated cancer specific survival for all patients combined was 86.6%, 74.0%, 68.7%, 63.8% and 60.0% at years 1, 3, 5, 7 and 10, respectively. However, independent factors were associated with prognosis [Table 1]. Therefore, prognosis could be better estimated by evaluating individual disease characteristics [Tables 2 and 3; Figure 1].

Table 1

Multivariable model for death from clear cell renal cell carcinoma. (Permission for reproduction obtained from Elsevier Publishing)

Table 2

Based on regression coefficients, prognostic score based on tumor characteristics. The SSIGN score is the sum of individual prognostic scores. (Permission for reproduction obtained from Elsevier Publishing

Table 3

Estimated cancer-specific survival following radical nephrectomy for clear cell renal cell carcinoma stratified by SSIGN score. (Permission for reproduction obtained from Elsevier Publishing)

Figure 1

Kaplan-Meier survival curves stratified by SSIGN score. (Permission for reproduction obtained from Elsevier Publishing)

How precise are the estimates of likelihood?

Intuitively, the more precise an estimate of prognosis is, the more useful it becomes.[2] Since estimates of prognosis are determined from a sample of patients, they should be considered estimates of the truth. Even in an ideal situation in which a study is free of bias and the sample is representative, the sample may not accurately reflect the truth in the population. This inherent risk of error due to chance necessitates a range of values around the best estimate in which the truth is likely to reside. This range is referred to as the confidence interval. The percentage associated with the interval, typically 95%, can be thought of as the degree of confidence we have that the range of values includes the true prognosis in the population if we repeated our study over and over. A narrow confidence interval indicates that the point estimate is more likely to be an accurate reflection of the true prognosis. Frank et al., did not include confidence intervals for each point estimate on their Kaplan-Meier survival curves; however, enough information is provided to calculate confidence intervals [Table 3]. Calculations of confidence intervals are beyond the scope of this article.

Utilizing the results for patient care

After completing a validity assessment and review of results you are ready to address the final question to determine if you should apply the study information to your patients.

Were the study patients and their management similar to your own?

A simple method to answer this question is to determine if your patient would have been included in the study and if the study patients were managed in a similar manner to your practice. Thankfully, your patient meets the inclusion criteria of the study and had a clear cell RCC managed by radical nephrectomy. However, since the study cohort was closed (1998) new systemic treatments are available that may prolong survival in patients with metastatic clear cell RCC.[910] Therefore, you believe contemporary patients may have prolonged survival compared to the study estimates.

Can I use the results in the management of patients in my practice?

The prognostic data you have acquired allows you to assess the effects of tumor characteristics on outcome, stratify patients for clinical trials, potentially use adjuvant therapies, and develop an appropriate postoperative surveillance program. You conclude that the results do assist you with patient management and are useful for reassuring and counseling patients.

Resolution of clinical scenario

You calculate your patient's SSIGN score is 4 (pT2 = 1; Tumor size >5 cm = 2; nuclear grade = 3) and determine his estimated cancer specific survival is 95.4%, 87.1%, 79.1%, 70.8%, 66.2% at years 1, 3, 5, 7 and 10, respectively. You call the patient and inform him that the probability of dying from RCC within 5 years is only 20%. You also inform him that this estimate may not be accurate and does not account for potential benefits of novel systemic treatments. He is somewhat relieved and assures you he will comply with cancer surveillance.

CONCLUSIONS

Prognostic studies provide data that can be used for research or patient care. As with all studies, prognostic study data is only useful if it is valid, transparent and applicable to your patients. Using the template provided we encourage readers to critically appraise prognostic studies and use the best available information to guide urology practice.