Defining the noninferiority margin and analysing noninferiority: An overview.

Noninferiority trials are used to assess whether the effect of a new drug is not worse than an active comparator by more than a noninferiority margin. If the difference between the new drug and the active comparator does not exceed this prespecified margin, noninferiority can be concluded. This margin must be specified based on clinical and statistical reasoning; however, it is considered as one of the most challenging steps in the design of noninferiority trials. Regulators recommend that the margin should be defined based on the historical evidence of the active comparator (the latter is often the well-established standard treatment of the disease), which can be performed by different approaches. There are several factors and assumptions that need to be accounted for during the process of defining the margin and during the analysis of noninferiority. Three methods are commonly used to analyse noninferiority trials: the fixed-margin method; the point-estimate method; and the synthesis method. This article provides an overview of analysing noninferiority and choosing the noninferiority margin.

Introduction

Randomized double‐blind placebo‐controlled trials are the gold standard for testing a new drug to treat a certain disease. These trials, if conducted properly, will show the intrinsic efficacy of the new drug and reveal the possible risk that might be associated with its use 1. However, the use of a placebo arm is unethical if it means denying patients an effective drug treatment 1, 2, 3, 4, 5. Instead, the new drug could be compared to the standard drug, which then acts as an active comparator. Such studies could have either superiority or noninferiority purposes.

Noninferiority trials seek to test whether the effect of the new drug is not unacceptably worse than the effect of the active comparator by more than a predefined noninferiority margin (often indicated by Δ). This is important when the new drug is believed to have a slightly better or slightly worse efficacy compared to the active comparator but offers safer, more cost‐effective or easier treatment options 1, 2, 3, 4, 5. This article provides an overview of the analysis of noninferiority trials and of the determination of a noninferiority margin. Important factors that contribute to the validity of the results of noninferiority trials are also described.

Analysing noninferiority

The analysis of noninferiority depends on the noninferiority margin that is the largest clinically acceptable difference between the test drug and the active comparator 1, 2, 3, 6, 7. There are several applications of the margin in the analysis of noninferiority trials, but the recommended approach by regulators, such as the US Food and Drug Administration (FDA), is to compare the estimated 95% confidence interval (CI) of the new drug vs. the active comparator from the noninferiority trial to a predefined margin 1, 2, 3, 6, 7. If the CI lies entirely below the margin (e.g. for effect measures where the larger the effect the worse the outcome), noninferiority of the new drug to the active comparator can be concluded. This then demonstrates that even if the difference was in favour of the active comparator, it did not exceed the unacceptably worse criteria of noninferiority (i.e. the noninferiority margin) 1, 2, 3, 6, 7. The Consolidated Standards of Reporting Trials (CONSORT) Statement for noninferiority and equivalence trials also recommends adding a figure showing where the CI lies with regard to the margin (such as the one showed in Figure 1) 8. Another, but rarely used, approach to analyse noninferiority is by testing the fraction of the effect of the active comparator that was retained by the new drug using a test statistic 2, 3, 9, 10, 11.

Figure 1

Analysing noninferiority by comparing the confidence interval (CI) of the relative risk to a predefined margin. (1) and (2) Noninferiority was not demonstrated because the upper limit of the CI exceeded the margin. (A), (B), (C) Noninferiority was demonstrated because the upper limits of the CI did not exceed the margin

Defining the noninferiority margin

Defining the noninferiority margin is crucial, yet one of the most challenging aspects in the design of noninferiority trials 2, 3, 4. Nevertheless, studies have shown that the method of determining the margin has not been mentioned in more than half of the published noninferiority trials 12, 13, 14, 15, 16, 17, 18. Regulators recommend that the margin should be defined based on statistical considerations and clinical judgement 1, 2, 6, 7. Statistical considerations are attributed to summarizing the historical evidence of the active comparator and, when possible, pooling an effect estimate with a 95% CI from the historical randomized controlled trials (mostly placebo‐controlled). The margin is defined either based on the pooled estimate or based on the limit of the CI that is the closest to the null effect (in either situation, both will be called M1). Clinical judgement is then applied to choose the fraction of M1 that must be preserved by the new drug (the preserved fraction). The margin represents the remaining fraction of M1, which is called M2. For example, if it is decided that 75% of M1 must be preserved by the new drug to demonstrate noninferiority, M2 = (1 – 0.75) × M1 = 0.25 × M1 (or 25% of M1) 2, 3.

Several factors govern the maximum loss of the effect of the active comparator that stakeholders are willing to accept in favour of the expected benefits of the new drug (i.e. choosing the preserved fraction). These factors include the seriousness of the outcome measure (e.g. irreversible morbidity or mortality), the effect size of the active comparator, the risk benefit‐profile and the cost of the active comparator, and whether it is believed that the effect of the active comparator has diminished over time 3, 4, 5, 11, 19. Snappinn and Jiang 11 and Snappinn 19 state that there are two possible interpretations for the preserved fraction. First, it represents a discounting in the effect of the active comparator to adjust for a possibly diminished effect over time. This adjustment is an approach to limit the bias due to the violation of one key assumption of noninferiority trials: the constancy assumption. The constancy assumption states that the effect of the active comparator in the noninferiority trial is the same (i.e., is constant) as in the historical studies 2, 3, 4, 5, 7, 10, 11, 19. Violation of this assumption may lead to a margin that is either too large or too small. For example, assume that a margin was defined based on a risk difference of 30% found in historical studies of the active comparator against placebo. Now assume that this effect has decreased over time, e.g. due to the improvement in the standard of care, so that now it would be only a risk difference of 10% if it was compared to placebo in a new trial. This means even a preserved fraction of 50% will lead to a margin of 15%, which exceeds the entire current effect of the active comparator against placebo, and noninferiority may be concluded even if the new drug is worse than (current) placebo. In this case, the objective of the noninferiority trial is to demonstrate that the effect of the new drug is indirectly superior to placebo, and the preservation of the effect acts as an additional assurance. The second interpretation, described by Snappinn and Jiang 11 and Sanppinn 19, which has been adopted by regulators, is that the preserved fraction acts as a threshold to demonstrate noninferiority (i.e. the effect of the new drug must be higher than the preserved fraction), and that the indirect superiority over a putative placebo is not sufficient.

A preserved fraction of 50% has become common practice in noninferiority trials (e.g. cardiovascular, irreversible morbidity, and mortality outcomes), but higher (i.e. stricter) fractions have been used (e.g. 90% preserved fraction in antibiotics) 2, 3, 12, 14, 17, 18. The stricter the preserved fraction, the harder to demonstrate noninferiority. This was demonstrated in the case‐study by Wangge et al. 20 about noninferiority trials of novel oral anticoagulants. In this case‐study, the fixed‐margin method was used to re‐analyse 16 noninferiority comparisons from 12 trials of new anticoagulants based on 50% and 67% preserved fractions. Enoxaparin was used as an active comparator in all these 16 comparisons. A discrepancy between the results of 50% and 67% analyses occurred in two out of 16 comparisons: two new anticoagulants were found to be inferior to the comparator when 67% preserved fraction was used instead of 50% preserved fraction.

Methods of analysing noninferiority

Using the noninferiority margin that was defined based on the historical evidence of the active comparator can be performed mainly by three methods: the fixed‐margin (95%–95% method); the point‐estimate; and the synthesis methods 2, 3, 9, 10. In all these three methods, the analysis of noninferiority is performed by comparing the CI from the noninferiority trial to the margin. The margin (M2) in the fixed‐margin method, which is the method that is recommended by the FDA, is conservatively defined based on the lower limit of the CI of the pooled point estimate that is closest to the null effect. This is considered as a secondary discount in the effect of the active comparator, beside the preserved fraction, to account for the uncertainty in the effect estimates of the active comparator from the historical trials and be conservative with respect to inferring noninferiority to protect against possible violation of the constancy assumption 2, 11, 19.

The margin in the point‐estimate and the synthesis methods is determined based on the pooled point estimate itself. In the point‐estimate method, it is assumed that the variability in the estimates of the active comparator is constant. In the synthesis method, the CI that was estimated from the noninferiority trial is adjusted to account for the variability of the estimates of the active comparator. The synthesis method is, however, often applied by determining a test statistic that shows whether the new drug retained a fraction (the preserved fraction) of the effect of the active comparator. This method was first described by Holmgren 9, and was later adopted by the FDA as a method to analyse noninferiority 2, 3, 10. The synthesis method could also be used to test whether the effect of the new drug is superior to a putative placebo 5, 21. Table 1 shows a list of noninferiority trials and the method used in each trial to analyse noninferiority. These trials also provide a good example on how the method of defining the margin should be reported.

Table 1

Analysis of noninferiority in published noninferiority trials

Trial	Treated condition	Test drug	Comparator	Outcome for noninferiority analysis	Noninferiority margin (and corresponding preserved fraction)	Method of analysis
SAVE‐ABDO 24	Thromboprophylaxis in patients with major abdominal surgery	Semuloparin (postoperatively)	Enoxaparin (preoperatively)	Composite endpoint of deep vein thrombosis, nonfatal pulmonary embolism, or all‐cause mortality	OR 1.25 (85%)	Fixed‐margin method
Natale et al. 25	Advanced nonsmall‐cell lung cancer	Vandetanib	Erlotinib	Progression‐free survival (PFS), overall survival (OR)	PFS 1.25 (50%)	Synthesis method
					OR 1.17 (50%)
Carbonell‐Estrany et al. 26	Prophylaxis of respiratory syncytial virus (RSV)	Motavizumab	Palivizumab	RSV hospitalization	RR 1.265 (50%)	Point‐estimate method
RE‐COVER 27	Acute venous thromboembolism	Dabigatran	Warfarin	Incidence of recurrent symptomatic, objectively confirmed venous thromboembolism, and related deaths	HR 2.75 (57%)	Fixed‐margin method
					RD 3.6% (75%)
PEARL 3 Ext 28	Chronic schizophrenia	Lurasidone	Quetiapine	Time‐to‐relapse of psychotic symptoms	HR 1.93 (50%)	Point‐estimate method

Hazard ratio, HR; relative risk, RR.

Examples of the analysis of noninferiority trials

The SPORTIF V (Stroke Prevention Using Oral Thrombin Inhibitor in Atrial Fibrillation V) trial is an example provided in the FDA guidance on the application of the fixed‐margin and the synthesis methods. The noninferiority of ximelagatran to warfarin was investigated in patients with nonvalvular atrial fibrillation to reduce the risk of thromboembolic complications. Using the fixed‐margin method, it was decided that ximelagatran should preserve at least 50% of the efficacy of warfarin, found in historical studies, to be considered noninferior. The relative risk (RR) of warfarin vs. placebo in the reduction of stroke and systemic embolism was 0.36 with a 95% CI of 0.25–0.53 based on six placebo‐controlled trials. The upper limit of this CI, which is the closest limit to the null effect, was used to define the margin. This limit represents a risk of 1.90 (1/0.53 = 1.90) of placebo compared to warfarin (90% increase in risk). In this example, M1 is 1.90 and M2 is 1.38 (50% of M1 on a logarithmic scale). Therefore, to conclude noninferiority, the upper limit of the relative risk of ximelagatran compared to warfarin must be <1.38. The relative risk of ximelagatran vs. warfarin in SPORTIF V trial was 1.39 (95%CI, 0.91–2.12). Since the upper limit of this CI exceeded the noninferiority margin, noninferiority of ximelagatran to warfarin was not demonstrated (Figure 2).

Figure 2

Analysing noninferiority of ximelagatran to warfarin using the relative risk. (1) Noninferiority margin for the fixed margin method. (2) Noninferiority margin for the point‐estimate and synthesis method. (*) The original confidence interval (CI) from SPORTIF V trial that was used to analyse noninferiority with the fixed‐margin and the point‐estimate methods. Noninferiority was not demonstrated with the fixed‐margin method and with the point‐estimate methods because the upper limit of the CI exceeded both margins (1.38 and 1.66). (&) The adjusted CI of SPORTIF V trial in the synthesis method. Noninferiority was not demonstrated because the upper limit of the CI is > the margin (1.66). (A), (B), (C) Noninferiority would have been demonstrated for all methods if the CI lies in one of the three positions in A, B or C

The conclusion did not change when we applied the point‐estimate method to reanalyse the SPORTIV V trial. The relative risk of placebo compared to warfarin from the six placebo‐controlled trials is 2.77 (1/0.36 = 2.77), and M2 is 1.66 (50% of the log relative risk of 2.77). Noninferiority of ximelagatran compared to warfarin was not concluded when using the point‐estimate method, because the upper limit of the CI of the risk of ximelagatran compared to warfarin in SPORTIV V trial (i.e., 2.12) exceeded the noninferiority margin of 1.66 (Figure 2). The same margin was used to apply the synthesis method, however, the CI was adjusted to account for the variability in the point estimates from the six placebo‐controlled trials of warfarin. The estimated standard error of the log(RR) of warfarin against placebo is 0.19, while the standard error of the log(RR) of ximelagatran against warfarin is 0.22. The indirect estimate of the standard error of the effect of ximelagatran against warfarin is . This leads to an indirect 95% CI around 1.39 of 0.87–2.22, which is wider than the CI based on the data of the noninferiority trial only. Again, the margin M2 is included in the CI and hence noninferiority cannot be concluded (Figure 2).

Another example comes from the Prevention Regimen for Effectively Avoiding Second Strokes (PRoFESS) trial, which was conducted to test the noninferiority of aspirin plus extended‐release dipyridamole twice daily to clopidogrel once daily for the prevention of recurrent strokes 22. Noninferiority in this trial was analysed by the fixed‐margin method with a hazard ratio of 1.08 as the noninferiority margin (M2). This margin corresponds to 50% of the lower limit of the 95% CI (1.16, M1) of placebo vs. clopidogrel that was estimated from historical studies (1.38; 95%CI, 1.16–1.65). Noninferiority was not demonstrated in this trial because the upper limit of the estimated CI of the combination therapy vs. clopidogrel exceeded the margin: 1.01, 95%CI, 0.92–1.11. However, the conclusion would have changed to noninferior if the point‐estimate or the synthesis methods were applied. In the point‐estimate method, the upper limit of the CI (1.11) did not exceed an alternative noninferiority margin (1.17, M2) based on 50% preserved fraction of the point estimate (1.38, M1) of the efficacy of clopidogrel compared to placebo. Hence, based on the point‐estimate method noninferiority could be concluded. When applying the synthesis method, the original CI of the effect estimate of the noninferiority study (95%CI, 0.92–1.11) is adjusted to account for the uncertainty of the estimates from the historical studies. In that case, the new upper limit of the CI (1.16) did not exceed the noninferiority margin (1.17), again demonstrating noninferiority.

Choosing the outcome measure to analyse noninferiority (absolute vs. relative metrics)

Absolute measures of effects, such as risk differences, tend to be more subject to heterogeneity in the effect estimates of the active comparator vs. placebo than relative measures of effects 2, 3, 5. To illustrate this, we analysed noninferiority in the SPORTIF V trial with the point‐estimate method using the risk difference. Firstly, we estimated a pooled risk difference (–3.75%; 95%CI, –5.54 to –1.96%) from the six placebo‐controlled trials of warfarin using the DerSimonian–Laird random‐effect model. This model was used due to the high level of heterogeneity among these six trials (I2 = 60%, Q = 12.4, p = 0.03). By choosing the point‐estimate method, the variability in the effect estimates from the six trials was considered constant (and was therefore ignored). The upper limit of the CI for the risk difference between ximelagatran and warfarin in SPORTIF V (risk difference: 0.72%, 95%CI –0.21 to 1.64%) did not exceed the noninferiority margin (1.88%, M2) that was defined to preserve 50% of the pooled effect of warfarin vs. placebo (3.75%, M1). Hence, noninferiority was concluded (Figure 3).

Figure 3

Analysing noninferiority of ximelagatran to warfarin using the risk difference. (1) Noninferiority margin for the fixed margin method. (2) Noninferiority margin for the point‐estimate and the synthesis method. (*) The original confidence interval (CI) from SPORTIF V trial that was used to analyse noninferiority with the fixed‐margin and the point‐estimate methods. Noninferiority was not demonstrated with the fixed‐margin method because the upper limit of the CI was > the margin (0.98%), whereas it was demonstrated with the point‐estimate because the upper limit of the confidence was < the margin (1.88%). (&) The adjusted CI of SPORTIF V trial in the synthesis method. Noninferiority was not demonstrated because the upper limit of the CI is > the margin (1.88%). (A), (B), (C) Noninferiority would have been demonstrated for all methods if the CI lies in one of the three positions in A, B or C

The large heterogeneity may be suggestive of violation of the constancy assumption. Indeed, the event rate of warfarin that was determined in SPORTIF V (1.2%) was half of the expected rate that was observed in historical studies (2.4%), suggesting violation of the constancy assumption 23.

The impact of this large heterogeneity of the risk differences from historical studies became evident when we performed a sensitivity analysis for the SPORTIF V trial using the historical placebo‐controlled trials of warfarin with low heterogeneity (i.e. only four of the six placebo‐controlled trials were included). Although the results of the analysis with the fixed‐margin and the synthesis methods were consistent compared to the results using all six trials, the conclusion changed when using the point‐estimate method. The new pooled estimate was –2.62% (95%CI, –3.77% to –1.47%) with no heterogeneity (I2 = 0%, Q = 0.73, P = 0.86). A noninferiority margin (1.31%, M2) was defined for the point‐estimate method to preserve 50% of the pooled point estimate (2.62%, M1). This margin was exceeded by the upper limit of the CI form SPORTIF V (1.64%), therefore, noninferiority was not demonstrated. By contrast, when using the relative risk, the conclusions of the sensitivity analyses, based on the four placebo‐controlled trials of warfarin, were consistent with the analyses performed using all six placebo‐controlled trials, irrespective of the methods of analysis used.

Conclusion

In this paper on noninferiority trials, we provided an overview of the considerations to define the noninferiority margin and the methods to analyse noninferiority. It is crucial for researchers, clinicians, and policy‐makers to understand these aspects to allow for optimal judgement of noninferiority trials.

Competing Interests

All authors have completed and submitted the International Committee of Medical Journal Editors Form for Disclosure of Potential Conflicts of Interest; and declare no support from any organization for the submitted work.

The Division of Pharmacoepidemiology & Clinical Pharmacology has received support for The PROTECT project from the Innovative Medicines Initiative Joint Undertaking (IMI JU) ( www.imi.europa.eu ) under Grant Agreement No. 115 004, resources of which are composed of financial contribution from the European Union's Seventh Framework Programme (FP7/2007–2013) and European Federation of Pharmaceutical Industries and Association companies' in‐kind contribution. As a special form of the IMI JU grant, Utrecht University received a direct financial contribution from Pfizer.

Funding/Support: This study was funded by the Saudi Food and Drug Authority (SFDA) as a part of a Doctor of Philosophy (PhD) project for Mr Althunian.

Role of the Funder/Sponsor: The SFDA has no role in any aspect of the study. Additionally, the SFDA has no role in the preparation, review, or the approval of the manuscript, and has no role in the publication of the manuscript.

Ethics approval: ethics approval was not required.