Assessing the benefit-risk of new treatments using generalised pairwise comparisons: the case of erlotinib in pancreatic cancer.

BACKGROUND: Efficacy and safety are the two considerations when characterising the effects of a new therapy. We sought to apply an innovative method of assessing the benefit-risk balance using data from a completed randomised controlled trial that compared erlotinib vs placebo added to gemcitabine in patients with advanced pancreatic cancer (NCIC CTG PA.3).
METHODS: We applied generalised pairwise comparisons with several prioritised outcome measures (e.g., one or more benefit outcomes and one or more risk outcomes). Here, the first priority outcome was overall survival (OS) time. Differences in OS that exceeded 2 months were considered clinically meaningful. The second priority outcome was toxicity. The overall treatment effect was quantified using the proportion in favour of erlotinib, which can be interpreted as the net proportion of patients who have a better overall outcome with erlotinib as compared with placebo. Sensitivity analyses were performed.
RESULTS: In this trial 569 patients were randomly assigned in a 1 : 1 ratio to receive gemcitabine plus either erlotinib or a matched placebo. Overall, the method indicated no statistically significant overall treatment effect in favour of erlotinib; if anything, the point estimate of the net proportion leaned in favour of the placebo group (overall proportion in favour of erlotinib=-3.6%, 95% CI, -14.2- 7.1%; P=0.51). The net proportion was never in favour of the erlotinib group throughout all sensitivity analyses.
CONCLUSIONS: Generalised pairwise comparisons make it possible to assess the benefit-risk balance of new treatments using a single statistical test for any number of prioritised outcomes. The benefit-risk assessment was not in favour of adding erlotinib to gemcitabine for the treatment of patients with advanced pancreatic cancer.

RCT Entities:   Population Interventions Outcomes

When characterising a treatment effect, efficacy and safety are the primary considerations. In the reporting of clinical trials, efficacy and safety outcomes are usually reported independently, no formal overall evaluation of the treatment effect is performed (Péron , 2013). Both US Food and Drug Administration and the European Medicines Agency have stressed the importance of a more structured and transparent approach to benefit–risk assessment (BRA) in the evaluation of new therapies (Committee for Medicinal Products for Human Use (CHMP), 2008; Food and Drug Administration, 2011).

Patients with advanced pancreatic cancer have a poor prognosis and the standard first-line regimen is cytotoxic chemotherapy (gemcitabine in monotherapy or in combination with nab-paclitaxel or a combination of 5-fluorouracil, oxaliplatin and irinotecan for patients with good performance status) (Burris ; Conroy ). The NCIC Clinical Trials Group Study PA.3 (NCIC CTG PA.3) phase III trial investigated the addition of erlotinib to gemcitabine in patients with advanced pancreatic cancer (Moore ). Both survival and progression-free survival were significantly better for the combination treatment but the overall benefits were of modest magnitude (HR for overall survival (OS)=0.82, 95% CI, 0.69–0.99; P=0.038). The excess toxicity, the unfavourable cost-effectiveness observed with the combination with erlotinib, (Miksad ; Tam ) and the absence of a biomarker predictive of erlotinib efficacy, (da Cunha Santos ; Boeck ) led to a poor uptake of this regimen in the oncology community (Verslype ; Saif, 2008; Choi ).

No systematic assessment of the benefit–risk balance of erlotinib combination has been performed in the setting of advanced pancreatic cancer. We report here such an assessment based on the method of generalised pairwise comparisons (Buyse, 2010). This method extends the non-parametric Mann–Whitney–Wilcoxon test for a single outcome in the absence of censored data. It allows one to calculate and test the overall benefit of a new treatment based on any number of prioritised outcomes, some reflecting benefit from the intervention (e.g., survival or time to progression) and the others reflecting harm (e.g., treatment-related toxicities and side effects).

Materials and methods

Overview

The NCIC CTG PA.3 trial was an international study that randomised patients with advanced pancreatic cancer to receive gemcitabine in combination with either erlotinib or placebo as first-line treatment. The primary outcome was OS. Progression-free survival (PFS) and toxicity were secondary outcomes.

In this trial, 569 patients were stratified by center, performance status (Eastern Cooperative Oncology Group 0 or 1 vs 2) and extent of disease (locally advanced vs metastatic), and randomly assigned in a 1 : 1 ratio to receive gemcitabine plus either erlotinib or a matched placebo. Progression was evaluated using Response Evaluation Criteria in Solid Tumors (V1.0) every 8 weeks. Toxicity was assessed at every visit using the National Cancer Institute Common Toxicity Criteria version 2.0.

Generalised pairwise comparisons

We applied generalised pairwise comparisons extended to several outcome measures (a benefit outcome, and a risk outcome). A full description of generalised pairwise comparisons has been previously published (Buyse, 2010). In brief, pairwise comparisons require consideration of all possible pairs of patients, one taken from the erlotinib arm and the other taken from the placebo arm. Pairwise comparisons are easily stratified for the stratification factors used in the randomisation process. The outcomes of these two patients are compared according to the first priority outcome. The pair is said to be ‘favourable' if the outcome of the patient in the erlotinib arm is better than the outcome of the patient in the placebo arm, ‘unfavourable' if the outcome of the patient in the erlotinib arm is worse than the outcome of the patient in the placebo arm and ‘uninformative' if it cannot be determined which of the two patients has a better outcome (e.g., because of censoring, because the two observations are equal or because the difference of outcomes does not reach a pre-specified threshold value). Such a pairwise comparison is carried out for all pairs of patients, and the difference between the proportion of favourable pairs and the proportion of unfavourable pairs is calculated for the first priority outcome. This difference is called the proportion in favour of treatment for the first priority outcome (Buyse, 2008; Moser and McCann, 2008).

For pairwise comparisons that are uninformative for the first priority outcome, the second priority outcome is used in turn to classify the pair as favourable, unfavourable or uninformative (Table 1). After consideration of the second priority outcome, the ‘overall proportion in favour of treatment' is calculated to provide an overall assessment of both the benefit and the risks of the treatment, suitably prioritised.

Table 1

Generalised pairwise comparisons for two prioritised outcomes

First priority outcome	Second priority outcome	Pair is
Favourable	Ignored	Favourable
Unfavourable	Ignored	Unfavourable
Uninformative	Favourable	Favourable
Uninformative	Unfavourable	Unfavourable
Uninformative	Uninformative	Uninformative

Standard analysis of efficacy and toxicity

A log-rank test adjusted for stratification factors at baseline was used to compare treatment groups in terms of survival. Worst grade adverse events (AE) that were at least possibly related to the study treatment (‘treatment-related AEs') were reported by treatment group. All analyses were performed on all randomly assigned patients as per the intent-to-treat principle.

Main analysis of the benefit–risk balance

The first priority outcome used in the main analysis was OS. Only pairs of patients with differences in OS exceeding 2 months were considered informative, because smaller differences in OS were not considered clinically meaningful. The second priority outcome was treatment-related AEs, with patients experiencing the lower grade-related AE considered to have had a more favourable outcome. Treatment arms were compared using the overall proportion in favour of the erlotinib group (Δ[erlotinib]). A randomisation test stratified by performance status and extent of disease at diagnosis was performed to test the null hypothesis (H0: Δ[erlotinib]=0). The contribution of each outcome to Δ[erlotinib] was calculated.

Sensitivity analyses

The impact of the choice of outcomes, thresholds and priority on the results was assessed in sensitivity analyses. First, the main analysis was repeated with various thresholds for the minimal OS difference considered as clinically meaningful, ranging from 0 (any difference in OS considered clinically meaningful) to 6 months. Second, the toxicity outcome was defined as a binary variable where only grade ⩾3 AEs were considered. Third, a subgroup analysis was performed among patients treated with 100 mg per day of erlotinib, the actual recommended dose. Finally, a wide range of scenarios integrating OS, PFS and AE grades with several successive thresholds were built to provide a comprehensive assessment of the treatment effects. For each scenario, the overall proportion in favour of the erlotinib group was calculated.

Results

Efficacy outcome

The main analysis of efficacy and safety was conducted after 486 deaths (239 on erlotinib and gemcitabine and 247 on placebo and gemcitabine) and has already been reported (Moore ). Overall survival was significantly longer in the erlotinib and gemcitabine arm with an estimated HR of 0.82 (95% CI, 0.69–0.99; P=0.011; log-rank test stratified for performance status, extent of disease). Median survival times were 6.24 months vs 5.91 months for the erlotinib and gemcitabine vs placebo and gemcitabine groups, respectively.

Four hundred and ninety-nine patients had developed progressive disease or had died at the end of the trial. Progression-free survival was significantly longer in the erlotinib and gemcitabine arm with an estimated HR of 0.77 (95% CI, 0.64–0.92; P=0.004; median, 3.75 months vs 3.55 months).

Toxicity outcomes

Two hundred eighty-two patients on the erlotinib and gemcitabine arm and 280 on the placebo and gemcitabine arm received at least one dose of study medication and were available for the assessment of toxicity.

The frequency of all grade and grade ⩾3 treatment-related AEs was higher for the erlotinib and gemcitabine group (90% and 31%, respectively) compared with the placebo and gemcitabine group (76% and 20%, respectively) (Table 2). The increase in grade ⩾3 AEs was especially notable for rash (6% vs 0%).

Table 2

Worst grade toxicity by treatment group

Worst grade related AE	Erlotinib group (n=282)	Placebo group (n=280)
Grade 1	48 (17.0%)	69 (24.6%)
Grade 2	118 (41.8%)	89 (31.8%)
Grade 3	72 (25.5%)	47 (16.8%)
Grade 4	11 (3.9%)	6 (2.1%)
Grade 5	4 (1.4%)	3 (1.1%)

Abbreviation: AE=adverse events.

Benefit–risk assessment

The proportion in favour of the erlotinib group was +4.7%, 95% CI, −5.6–14·6% (thus favouring erlotinib) for the first priority outcome (OS) but −8.3%, 95% CI, −14.2–7.1% (thus favouring placebo) for the second priority outcome (toxicity) among patients uninformative on the OS outcome. Overall, the net proportion favoured non-significantly the placebo group (overall Δ[erlotinib]=−3.6, 95% CI, −14.2– 7.1; P=0.51), suggesting an unfavourable benefit–risk balance of erlotinib added to gemcitabine (Table 3).

Table 3

Main analysis of the benefit–risk balance of erlotinib and gemcitabine combination

	Proportion of pairs (%)		Difference
Priority	Erlotinib>placebo	Placebo>erlotinib	Δ[erlotinib]
OS (threshold=2 months)	37.0	32.3	4.7
Worst related AE grade	7.5	15.7	−8.3
Overall	44.5	48.1	−3.6 (P=0.51)

Abbreviations: >=better than; AE=adverse events; Δ[erlotinib]=proportion in favour of the erlotinib group; OS=overall survival.

The analysis was repeated with various values for the OS threshold, varying between 0 and 6 months. When the OS threshold was set at 0 month, meaning that any difference in OS was considered meaningful, the overall analysis was not statistically significant (overall proportion in favour of erlotinib=2.3, 95% CI, −8.1–12.7; P=0.67). This setting gave a large weight to the first priority OS outcome, because any survival improvement was considered clinically significant, regardless of AEs. As the OS threshold increased, the overall assessment leaned more and more in favour of the placebo group. It reached statistical significance in favour of erlotinib for values of the OS threshold >5 months (Figure 1).

Figure 1

Benefit–risk of erlotinib according to the minimum survival benefit considered clinically meaningful. Proportion in favour of erlotinib according to the minimum survival benefit considered clinically meaningful. First priority outcome: overall survival. Second priority outcome: worst grade of at least possibly related adverse events. Solid black line with asterisks: proportion in favour of erlotinib according to the first priority outcome (OS) only. Solid light-grey line with points: overall proportion in favour of erlotinib.

The analysis was repeated using a threshold of two AE grades for the second priority toxicity outcome (hence, in this analysis, a difference of one grade or less was not considered clinically meaningful). Again, the analysis tended to favour the placebo group but remained non-significant statistically (Table 4).

Table 4

Sensitivity analysis of the benefit–risk balance of the erlotinib and gemcitabine combination – only differences in treatment-related AEs of at least two grades are considered clinically meaningful

	Proportion of pairs (%)		Difference
Priority	Erlotinib>placebo	Placebo>erlotinib	Δ[erlotinib]
OS (threshold=2 months)	37.0	32.3	4.7
Worst related AE grade (threshold=2 grades)	3.0	8.4	−5.3
Overall	40.1	40.7	−0.6 (P=0.90)

Abbreviations: >=better than; AE=adverse events; Δ[erlotinib]=proportion in favour of the erlotinib group; OS=overall survival.

When only Grade ⩾3 AEs were considered in the second priority toxicity outcome, the overall proportion in favour of erlotinib was again low for OS threshold under 2 months (+1.5, 95% CI, −8.5–11.4; P=0.77) and became negative for OS thresholds larger than 2.5 months (Figure 2). The analyses never reached statistical significance for the tested OS thresholds (up to 6 months).

Figure 2

Benefit–risk of erlotinib according to the minimum survival benefit considered clinically meaningful. Only grade 3–4 adverse events are included in this analysis. Proportion in favour of erlotinib according to the minimum survival benefit considered clinically meaningful. First priority outcome: overall survival. Second priority outcome: presence of a grade 3–4 at least possibly related adverse events. Solid black line with asterisks: proportion in favour of erlotinib according to the first priority (OS) outcome only. Solid light-grey line with points: overall proportion in favour of erlotinib.

When skin rashes were excluded from the list of AEs analyzed in the second priority outcome, the overall analysis was not in favour of erlotinib (overall proportion in favour of erlotinib=−0.3, 95% CI, −9.1–8.4; P=0.94) (Table 5). A subgroup analysis was performed according to the occurrence of a grade ⩾2 rash in the erlotinib group. The benefit–risk of erlotinib in the subgroup of patients experiencing grade ⩾2 rashes was statistically significantly favourable (Δ[erlotinib]=13.7; P=0.032), and it was statistically significantly unfavourable in the subgroup of patients with grade 0 or 1 rashes (Δ[erlotinib]=−13.8; P=0.016) (Table 6).

Table 5

Sensitivity analysis of the benefit–risk balance of the erlotinib and gemcitabine combination – skin rashes are excluded from the list of adverse events

	Proportion of pairs (%)		Difference
Priority	Erlotinib>placebo	Placebo>erlotinib	Δ[erlotinib]
OS (threshold=2 months)	37.0	32.3	4.7
Worst related AE gradea	9.1	14.1	−5.0
Overall	46.1	46.4	−0.3 (P=0.94)

Abbreviations: >=better than; AE=adverse events; Δ[erlotinib]=proportion in favour of the erlotinib group; OS=overall survival.

Skin rashes are excluded from the list of adverse events.

Table 6

Analysis of the benefit–risk balance of the erlotinib and gemcitabine combination, according to the occurrence of a grade⩾2 rash in the erlotinib group

	Δ[erlotinib]
Priority	Grade ⩾2 rash in the erlotinib group	Grade 0–1 rash in the erlotinib group
OS (threshold=2 months)	31.2	−11.0
Worst related AE grade	−17.5	−2.8
Overall	13.7 (P=0.032)	−13.8 (P=0.016)

Abbreviations: AE=adverse events; Δ[erlotinib]=proportion in favour of the erlotinib group; OS=overall survival.

In the subgroup of the 521 patients treated with 100 mg per day of erlotinib, the main analysis of benefit–risk once again was not in favour of the erlotinib (overall proportion in favour of erlotinib=−2.7, 95% CI, −13.6–8.1; P=0.62).

Comprehensive sensitivity analyses of the benefit–risk were carried out using various thresholds for OS, PFS and worst AE grade. Some scenarios with clinically meaningful choices of end point prioritisation and of thresholds are presented in Table 7. For none of the scenarios considered was the overall benefit risk assessment in favour of erlotinib.

Table 7

Further sensitivity analyses of the benefit–risk balance of the erlotinib and gemcitabine combination, using different priorities and threshold values for the outcomes of interest

		Proportion of pairs (%)		Difference
Priority	Threshold	Erlotinib>placebo	Placebo>erlotinib	Δ[erlotinib]
1. OS	6 months	16.8	13.1	3.6
2. PFS	6 months	3.0	1.8	1.2
3. Worst related AE gradea	3 grades	2.3	5.8	−3.5
4. OS	3 months	11.2	10.8	0.4
5. PFS	3 months	3.4	2.7	0.7
6. Worst related AE gradea	2 grades	2.3	6.0	−3.7
7. OS	0 months	9.5	9.1	0.4
8. PFS	0 months	0.5	0.6	−0.1
9. Worst related AE gradea	1 grade	0.2	0.5	−0.3
Overall		49.2	50.4	−1.2 (P=0.82)
1. OS	4 months	25.7	21.8	3.9
2. PFS	4 months	4.5	2.6	1.9
3. Worst related AE gradea	2 grades	5.0	11.9	−6.9
4. OS	2 months	6.1	5.8	0.3
5. PFS	2 months	2.0	1.9	0.1
6. Worst related AE gradea	1 grade	3.1	4.7	−1.6
7. OS	0 months	2.1	2.1	0.0
8. PFS	0 months	0.2	0.2	0.0
Overall		48.7	50.9	−2.2 (P=0.67)
1. Worst related AE gradea	3 grades	3.2	8.8	−5.6
2. OS	4 months	22.8	18.9	3.9
3. PFS	4 months	4.0	2.5	1.6
4. Worst related AE gradea	2 grades	3.3	7.8	−4.5
5. OS	2 months	6.1	5.8	0.3
6. PFS	2 months	2.0	1.9	0.1
7. Worst related AE gradea	1 grade	3.1	4.7	−1.6
8. OS	0 months	2.1	2.1	0.0
9. PFS	0 months	0.2	0.2	0.0
Overall		46.9	52.7	−5.8 (P=0.27)

Abbreviations: >=better than; AE=adverse events; Δ[erlotinib]=proportion in favour of the erlotinib group; OS=overall survival; PFS=progression-free survival.

Only differences beyond the threshold value in treatment-related AEs are included in this toxicity assessment.

Discussion

We have used generalised pairwise comparisons, prioritised on several outcomes, to perform an assessment of the benefit–risk balance of adding erlotinib to gemcitabine for the treatment of patients with advanced pancreatic cancer. These analyses showed that the OS benefit in favour of erlotinib diminished when using increased thresholds for the OS benefit and/or adding AEs in an assessment of the net benefit of the combination. The benefit risk assessment did not favour adding erlotinib in the main analysis, and this result was confirmed in all sensitivity analyses.

The method of generalised pairwise comparisons gives higher priority to the outcome considered clinically more important – in this case, overall survival was considered more important than any grade of toxicity. The method can incorporate both a priority and a threshold for each of the outcomes considered (in this instance, OS and treatment-related toxicities), and as such it reflects the thinking process of clinicians and decision makers, who try to assess the net effect of a new treatment on several outcomes considered to be of clinical importance. As such, the method may be particularly informative in health technology assessment.

Several methods have been proposed to help the scientific assessment of the benefit–risk balance of interventions. These methods are most frequently designed to weigh relevant efficacy and safety data into a single construct (Committee for Medicinal Products for Human Use (CHMP), 2008). QALY is a measurement of health status that assigns a weight in each period of time according to the quality of life during this period (Weinstein ). It might be used to adjust a gain in survival to an increased level of toxicity by assigning a smallest weight to the time of survival with significant toxicity. However, it requires clearly defined health states, as well as weights for each state, which might be difficult to establish when planning a trial. This limitation makes QALY difficult to use as a primary end point to evaluate therapeutic interventions, and a more suitable tool for medico-economic evaluation (Whitehead and Ali, 2010). Other methods such as Overall Treatment Utility (OTU) can be used to combine subjective and objective measures of the treatment effect into a single composite end point. However the respective weights of the different treatment effects included in OTU may be difficult to justify and to report (Seymour ).

The method of generalised pairwise comparisons only requires the priority of each outcome to be defined. Sensitivity analyses are useful to confirm the conclusion of the main analysis. Indeed, the conclusion may rest entirely on arbitrary (though arguably relevant) choices made regarding outcome priorities and thresholds values (if any). Most clinicians and patients would agree that small gains in survival cannot be considered as a positive outcome if such gains are obtained at the expense of severe toxicities. However, determining the minimal survival benefit threshold for which most patients would accept to experience a treatment-related AE is very complex. It may depend on the type of AE and its grade, and it may vary considerably from patient to patient. Survival benefits may be offset by severe and/or long-term AEs. Investigators can now use generalised pairwise comparisons to test the benefit–risk balance of investigational therapies, depending on the level of tolerable toxicity that is deemed acceptable for a given magnitude of survival benefit. Various scenarios for the threshold of survival benefit and the grades of AEs are reported in the Table 7. Throughout all the scenarios, the benefit–risk balance leaned against erlotinib, which does provide some confirmation of the results of the main analysis. Moreover the clinical impact of AEs may vary a lot depending of the type of AEs, even among AEs of the same grade. When skin rashes were excluded from the list of relevant adverse events, the benefit risk assessment of erlotinib was close to zero.

Relevant toxicity criteria could potentially vary from trial to trial. For example, a risk assessment could focus on predefined AEs of special interest, on all severe AEs, on severe treatment-related AEs, or on AEs leading to drug discontinuation. (Ioannidis ) For the PA.3 trial, the frequency of lethal AEs or of AEs leading to treatment discontinuation was low, as well as the frequency of grade 3–4 AEs.

Generalised pairwise comparisons are useful to perform a quantitative assessment of the benefit–risk balance of a new treatment as compared with a standard therapy. Such an assessment is especially useful when overall efficacy differences are small, and no subset of patients has been identified as being more likely to benefit from treatment. In such cases, generalised pairwise comparisons provide a clinically intuitive way of comparing patients with respect to all important efficacy and toxicity outcomes, with full flexibility as to the priority of each outcome, and a threshold of clinical significance. In particular, when some patients benefit from treatment at the price of a given toxicity (e.g., severe treatment-related rash after administration of a tyrosine kinase inhibitor), the prioritisation of their outcomes naturally ensures that the benefit trumps the toxicity in the overall assessment of the benefit–risk balance.