Unbiased estimation in seamless phase II/III trials with unequal treatment effect variances and hypothesis-driven selection rules.

Seamless phase II/III clinical trials offer an efficient way to select an experimental treatment and perform confirmatory analysis within a single trial. However, combining the data from both stages in the final analysis can induce bias into the estimates of treatment effects. Methods for bias adjustment developed thus far have made restrictive assumptions about the design and selection rules followed. In order to address these shortcomings, we apply recent methodological advances to derive the uniformly minimum variance conditionally unbiased estimator for two-stage seamless phase II/III trials. Our framework allows for the precision of the treatment arm estimates to take arbitrary values, can be utilised for all treatments that are taken forward to phase III and is applicable when the decision to select or drop treatment arms is driven by a multiplicity-adjusted hypothesis testing procedure.

Introduction

Seamless phase II/III designs are key examples of adaptive clinical trials where data from a learning phase and a confirmatory phase are combined to promote efficient drug development. Typically, such trials will have two stages separated by an interim analysis. In stage 1, which resembles a traditional phase II trial, multiple experimental treatments or drug doses are simultaneously compared against a control. The most promising candidates are then selected for confirmatory analysis in stage 2, which corresponds to a phase III trial.

Recent examples of seamless phase II/III trials in clinical practice include dose selection for chronic obstructive pulmonary disorder 1, acute myocardial infarction 2 and treatment selection for colorectal cancer 3. Regulatory guidance dealing with such adaptive trial designs has been produced in Europe by the European Medicines Agency 4 and in the USA by the Food and Drug Administration 5.

Unlike the classical approach where only phase III patients contribute to the confirmatory analysis, in seamless phase II/III trials, the final analysis utilises data from both stages. Whilst combining the data is efficient in terms of time and resources (and of course in a purely statistical sense too), it can inflate the type I error of hypothesis tests and induce bias into the naïve estimates of treatment effect, because of the dual influence of multiplicity and selection 6.

In this paper, we consider point estimation of the treatment effects. In particular, our focus is on conditionally unbiased estimation using the method of Rao–Blackwellization. Briefly, this involves taking the unbiased stage 2 data and conditioning on a complete, sufficient statistic. The resulting estimator is the uniformly minimum variance conditionally unbiased estimator (UMVCUE).

Our key starting point in deriving the UMVCUE for the treatment effect is the paper by Kimani et al. 7. Building on the seminal framework of Cohen and Sackorwitz 8, Kimani et al. derived the UMVCUEs for the means of the selected and control treatments separately and then took the difference to give an unbiased estimator for the treatment difference. However, a number of limitations to this approach currently exist.

Firstly, the methodology does not try to explicitly take into account differing treatment effect variances. Hence, if treatment selection is based on standardised differences, the estimator will not necessarily be unbiased when the treatment effect variances are unequal. Many other authors have also used the convention of equal variances in order to derive their results 9, 10, 11, 12. This allows the information from the treatment and control arms at phases II and III to be separated, and all estimates can be assumed to be independent. However, there are many reasons why differences in the treatment effect variances may occur, even if this was not planned at the outset. For example, there may be unequal drop out across arms, or there simply may be true differences between the variance of patient outcomes for different experimental treatments.

Secondly, the estimator is only for the treatment with the largest treatment difference. However, we may be interested in estimating the treatment difference for several treatments, for example, when the decision to select/drop treatments is driven by a formal hypothesis testing procedure. We note that formulae that are applicable to this setting have been derived by Bowden and Glimm 10, but for different ranking and selection rules to those considered in this paper. In practice, and as for any confirmatory trial, hypothesis testing (with rigid type I error control) will be the primary focus of a seamless phase II/III trial, with estimation being an important but secondary target. For a comprehensive overview of the methodology for hypothesis testing in seamless phase II/III trials, we refer the reader to the reviews of Bretz et al. 13 and Stallard and Todd 14.

In this paper, we aim to address these limitations, by transferring recent methodological advancements in UMVCUEs for multivariate normal outcomes proposed by Robertson et al. 15 to the seamless phase II/III setting. We derive formulae that are applicable in full generality for the jth‐ranked treatment where the precision of treatment arm estimates can take arbitrary values.

The rest of the paper is organised as follows. In Section 2, we describe the set‐up and notation, derive the UMVCUE for the maximum treatment difference and compare it analytically with the Kimani et al. estimator. We carry out a simulation study in Section 3 to compare the bias and mean square error of the Kimani et al. estimator and our UMVCUE in a variety of trial settings. Section 4 describes how our UMVCUE can be used in the context of a seamless phase II/III trial where a multiplicity adjusted hypothesis procedure drives the design and is illustrated with a simple practical example. We discuss all of our results in Section 5 and consider future avenues of research.

Framework for the uniformly minimum variance conditionally unbiased estimator

We use the adaptive seamless design (ASD) setting of Kimani et al. 7 as our starting point. Consider an ASD with two stages, where stage 1 is used to select the most promising treatment and stage 2 is used for confirmatory analysis. Let K( ) denote the number of experimental treatments tested in stage 1 for comparison with the control. The treatment that shows the highest standardised treatment difference (as defined in the succeeding text) in stage 1 is then selected to continue to stage 2, along with the control.

We now allow for the treatment arm estimates to have unequal variances. Let n 1 denote the number of subjects allocated to treatment i (i = 0,1,…,K) in stage 1, where i = 0 corresponds to the control treatment. We assume that the stage 1 sample mean for treatment i, denoted X , is normally distributed with unknown mean μ and known variance . As an example, if we also assume that there is a known common variance σ 2 across the treatment groups, then .

At the end of stage 1, we rank the treatments according to their standardised treatment difference. More explicitly, we rank treatment i above treatment j if

In contrast, in the Kimani et al. setting, we rank the treatments by the stage 1 sample means, and so treatment i is ranked above treatment j if X >X . Note that if we have a common stage 1 variance, that is, σ 1=σ 1 for i = 1,…,K, then the two ranking rules are the same, because when ranking by standardised treatment difference, the denominator and control data X 0 can be ignored.

We let the treatment with the highest ranking be denoted S (S∈{1,…,K}), where S is a random variable. We also allow early stopping of the trial for futility: the trial continues to stage 2 if , where b is a (pre‐specified and constant) futility boundary.

For notational convenience, let Θ=X −X 0 denote the stage 1 sample mean treatment difference for treatment i (i = 1,…,K) and define (i = 1,…,K). Then treatment i is ranked above treatment j if λ Θ>λ Θ. As well, the futility boundary implies that λ Θ>b in order for the trial to continue to stage 2.

If the trial continues to stage 2, then let n 2 denote the number of subjects allocated to treatment i (i = 0,S). We assume that the stage 2 sample means, denoted Y , follow a distribution. As before, if we also assume a known common variance σ 2, then . We can define the selection time for treatment i (i = 0,S) as . Hence, the sample mean from the two stages for the control is Z 0,MLE=t 0 X 0+(1 − t 0)Y 0 and similarly Z =t X +(1 − t )Y for the selected treatment.

After the trial is completed, the aim is to estimate the treatment difference θ =μ −μ 0. As Kimani et al. note, the maximum likelihood estimator (MLE) for θ is D =Z −Z 0,MLE. This estimator will likely be biased, because it does not take into account the selection rules. An unbiased estimator can easily be found by just using the stage 2 data, because Y and Y 0 are unbiased estimators for μ and μ 0, respectively. Hence, the sample difference Y = Y −Y 0 is an unbiased estimator for θ . However, this estimator will be inefficient because it does not use the stage 1 data.

Calculating the uniformly minimum variance conditionally unbiased estimator

Using the theory from the general multivariate normal setting 15, we derive the UMVCUE for this framework. The stage 1 sample mean treatment differences Θ=X −X 0 are normally distributed: . Because Θ = (Θ1,…,Θ) is a linear transformation of = (X 0,X 1,…,X ), then Θ follows a multivariate normal distribution with mean = (θ 1,…,θ ) and covariance matrix Σ, where θ =μ −μ 0 and Σ=Cov(Θ,Θ). Hence, The stage 2 sample mean treatment difference Y = Y −Y 0 is also normally distributed with . Let Q be the event {Θ:λ 1Θ1>λ 2Θ2>⋯>λ Θ,λ 1Θ1>b}, which implies that the trial continues to stage 2 and that treatment i has rank i, with S = 1. Without loss of generality, we condition on Q for the remainder of this section. For notational convenience, let and . Then the statistics = (Z 1,…,Z ) are sufficient and complete for , where

Using the notation defined previously, we have the following form for the UMVCUE, with a proof provided in Appendix A.1.

The UMVCUE for θ 1=μ 1−μ 0 given Q is where and we define min({∅}) =+ ∞ and max({∅}) =− ∞.

Note that the first term in expression (3), namely, , is equal to the MLE D .

Comparison with the estimator of Kimani et al.

Suppose we set (for i = 0,1,…,K) and (for i = 0,1). We then recover the setting of Kimani et al. 7, so we can compare our results. In this case, ranking by standardised treatment difference reduces down to ranking by the stage 1 sample mean, in the sense that they always select the same treatment. Kimani et al. derived the following unbiased estimator for θ 1=μ 1−μ 0: where As for the UMVCUE derived in Section 2, firstly note that σ 1,=σ 1=σ 1 for j = 2,…,K − 1. Hence, the sets A 3 and A 5 are empty in Equation (3). In addition, , and hence, A 4 is also empty. Note also that and .

The sufficient statistics in this case are Hence, the UMVCUE equals where Now, we can rewrite W as Even for the special case when the two methods always select the same treatment, the estimators are not equal, because the estimators condition on different selection rules and data. We return to this issue in Section 3.1.1.

Simulation study

We now perform a simulation study to explore the bias and mean squared error (MSE) of the estimators described in Section 2. Because the performance of the Kimani et al. (D 1,CHN), naïve (D 1,MLE) and stage 2 (D 1,2) estimators have already been extensively studied in 7, we focus on comparing the properties of our UMVCUE with these existing estimators.

Equal variances

Initially we use the setting of Kimani et al. 7, with a common variance σ 2, n 1=n 1(i = 0,1,…,K) and n 2=n 2(i = 0,1). Hence, the stages 1 and 2 variances are all equal, and we can write and . Also the selection times t all equal .

In our simulations, we set the common variance σ = 1 and vary the selection time point t in the interval (0,1). Because the stages 1 and 2 sample sizes per arm are equal, we can present the bias and the of the estimators in units of the standard error (SE) . This is the standard deviation for the difference of a single experimental treatment–control comparison and makes the results invariant to sample size 7.

Figure 1 shows the when the number of experimental treatments K = 2 and μ 0=0,μ 1=μ 2=0.05. We assume there is no early stopping for futility, which corresponds to the futility boundary b =− ∞. Note that we do not give a plot of the bias because (as expected) the bias of our UMVCUE (as well as the Kimani estimator) is not noticeably different from zero in the simulations. The MSE of the Kimani estimator and UMVCUE are approximately equal, but for all values of t, the UMVCUE has a higher MSE. This difference is an increasing function of t.

Figure 1

for various estimators, in units of standard error (SE). We set μ 0=0,μ 1=μ 2=0.05 and b =− ∞. There were 20000 simulated trials for each value of the selection time t. MSE, mean squared error; UMVCUE, uniformly minimum variance conditionally unbiased estimator.

Table 1 shows the bias and (in units of SE) for a range of representative parameter values, with the selection time t = 0.5. As expected, the UMVCUE is unbiased in its mean in all cases. The UMVCUE still has a slightly higher – although it is within 10% of the for the Kimani estimator. This difference is a decreasing function of the futility boundary b (or equivalently, an increasing function of the probability of early stopping for futility).

Table 1

Simulation results for t = 0.5. There were 100000 simulations for each set of parameter values.

	Bias MSE in units of SE
Parameter values	Naïve	Stage 2	Kimani	UMVCUE
μ 0=0,μ 1=μ 2=0.05	0.286	0.002	0.003	0.003
b =− ∞	(1.002)	(1.412)	(1.085)	(1.119)
μ 0=0,μ 1=μ 2=0.05	0.511	−0.008	−0.003	−0.004
b = 0	(1.002)	(1.407)	(1.188)	(1.198)
μ 0=0.1,μ 1=μ 2=0.3	0.276	−0.005	−0.007	−0.006
b =− ∞	(0.997)	(1.419)	(1.083)	(1.119)
μ 0=0.1,μ 1=μ 2=0.3	0.330	0.004	0.004	0.005
b = 0.1	(0.986)	(1.413)	(1.111)	(1.140)
μ 0=0.05,μ 1=0.15,μ 2=0.1	0.439	−0.004	−0.003	−0.004
b = 0	(0.985)	(1.414)	(1.166)	(1.181)
μ 0=0,μ 1=μ 2=μ 3=μ 4=0.05	0.650	0.005	0.005	0.003
b = 0.05	(1.087)	(1.412)	(1.186)	(1.222)

MSE, mean squared error; UMVCUE, uniformly minimum variance conditionally unbiased estimator.

How can the UMVCUE be worse in terms of MSE?

The result that the UMVCUE has a higher MSE than the Kimani estimator seems somewhat counter‐intuitive and indeed seemingly in contradiction of the very definition of the UMVCUE we have derived. However, the explanation is that the two estimators are using different amounts of data. The Kimani estimator is a function of the individual treatment mean outcome statistics X 0,X 1,…,X ,Y 0,Y 1, whereas our UMVCUE is a function of X 1−X 0,…,X −X 0,Y 1−Y 0. That is, we are not explicitly using the control data X 0,Y 0 in the UMVCUE – all we need are the treatment differences in both stages. In the special case of equal variances, for which the experimental and control group data can be separated, this loss of information results in a slightly greater MSE for the UMVCUE. We return to this issue in the discussion.

Unequal variances

We have seen that the Kimani estimator performs well when the stages 1 and 2 variances are equal. We now explore what happens when this assumption no longer holds – that is, when the σ 1 and σ 2 are distinct. In this setting, ranking by standardised treatment difference no longer reduces down to ranking by the stage 1 sample mean. This means that the selection based on standardised observed differences will not necessarily select the treatment with the highest stage 1 sample mean. The Kimani estimator will overcorrect for bias in this setting, because it assumes that we are always selecting the treatment with the highest treatment effect.

We now conduct simulation studies to see to what extent the Kimani estimator is appropriate for the selection rule that uses standardised treatment differences. Although the Kimani estimator is being incorrectly applied in this setting, because it slightly outperformed the UMVCUE in terms of MSE when the variances are equal, it is interesting to investigate whether it does so again.

We can straightforwardly modify the Kimani estimator to take into account the differing variances in stages 1 and 2 as follows: where Consider now the scenario where K = 2 and one of the experimental treatments has variance say, whereas the other experimental treatment and the control both have variance equal to 1. Figure 2 shows the (unadjusted) bias and for the various estimators where we vary from 0.25 to 4. We also set σ 21=σ 11, whilst keeping σ 20=1. Note that is the variance of the treatment that is selected to continue to stage 2.

Figure 2

Bias and for the estimators, using individual variances for the Kimani estimator. We set μ 0=0, μ 1=μ 2=0.05, σ 21=σ 11 and σ 10=σ 20=1. There were 50000 simulated trials for each value of .

If we assume a common variance σ 2 across treatment groups, then values of (or <0.5) imply unrealistic unequal allocations to the treatment groups that would rarely occur in practice. However, such scenarios could occur where there is reason to believe treatment 1 has a different treatment effect variance from the other treatment (and the control) after looking at previous trial or pilot study data. This may make biological sense too, if treatment 1 is a different class of drug to the others. In that case, we could have despite having equal allocation ratios to the treatment groups and the control.

As expected, the stage 2 estimator and UMVCUE are unbiased for all values of , whilst the naïve estimator is positively biased. However, we see that for , the Kimani estimator is negatively biased, with the bias much worse when b =− ∞ compared with b = 0.1. This negative bias steadily increases as increases above 1. Indeed, when , the Kimani estimator has a substantial bias when b =− ∞(which corresponds to an early stopping probability of 0).

In terms of the MSE, as expected the naïve estimator has the lowest MSE. The Kimani estimator has a higher MSE than the UMVCUE except when is close to 1. There is a steady increase in the Kimani estimator's MSE for . The MSE of the UMVCUE is slightly higher when b = 0.1 compared with b =− ∞ and vice‐versa for the MSE of the Kimani estimator.

Unbiased estimation for hypothesis‐driven designs

Finally, we look at the application of our estimator within the context of formal hypothesis testing. We illustrate this with an example based on the case study in 16. Suppose we are comparing three experimental drugs with a placebo for the treatment of generalised anxiety disorder. We assume that the outcomes (the total score on the Hamilton Rating Scale for Anxiety) are normally distributed with common standard deviation σ = 6.

The trial is planned with equal allocations to each treatment, with n 1=n 2=71 subjects per group. However, suppose that the randomisation procedure used leads to an unequal number of subjects in each treatment group. Table 2 shows the observed data for both stages of the trial.

Table 2

Example data from a seamless phase II/III trial.

	Stage 1					Stage 2
	n 1i	Observed	z‐statistic	p 1i		n 2i	Observed
Placebo	70	0.4	—	—		68	−0.3
Treatment 1	72	2.2	1.787	0.0369		75	1.7
Treatment 2	68	2.4	1.958	0.0251		70	2.2
Treatment 3	74	3.2	2.799	0.0026		71	1.9

The aim is to take forward as many treatments as possible that pass a first‐stage p‐value futility threshold, set at α 0=0.1. As we are in a multiple testing situation, we use multiplicity corrected p‐values and the closure principle in our analysis (see Section 4.2). Although the primary focus will be hypothesis testing, estimation of the treatment effects is an important secondary goal, and we would like unbiased estimates of the selected treatments' benefit over control at the end of the trial. This means that we need a way of estimating the treatment difference when (i) multiple treatments are taken forward to stage 2, and (ii) the treatments are not ranked using a rule that is concordant with ranking by the stage 1 sample mean alone. In these cases, the Kimani estimator cannot be used, and hence we need to extend our methodology (see succeeding text).

Uniformly minimum variance conditionally unbiased estimator for the jth‐ranked treatment

Suppose that we take forward the top K treatments from a larger group of K′. We want to find the UMVCUE for the jth best treatment out of K. Let T =Y −Y 0 denote the stage 2 sample mean treatment differences for j∈{1,…,K}. We consider the more general early stopping rules where the jth‐ranked treatment proceeds to stage 2 if λ (X −X 0) > b .

For notational convenience, let and . Then from the multivariate normal theory 15, for a given value of j∈{1,…,K} the statistic = (Z 1,…,Z ) is sufficient and complete for , where This time, we are conditioning on the modified event Q′, where Q′={Θ:λ 1Θ1>⋯>λ Θ,λ 1Θ1>b 1,…,λ Θ>b }. Then the UMVCUE for the jth‐ranked treatment (denoted  ) is shown in Theorem 4.1 with a proof provided in Appendix A.2.

For a given value of j∈{1,…,K}, the UMVCUE for θ =μ −μ 0 given Q′ is where and we define min({∅}) =+ ∞ and max({∅}) =− ∞.

Uniformly minimum variance conditionally unbiased estimator with the closure principle

We can now start to apply our UMVCUE to the example trial setting. In general, assume that we are testing the K directional null hypotheses (i = 1,…,K) comparing the K treatments with the control. Our aim is to strongly control the familywise error rate (FWER) at a pre‐specified level α, where strong FWER is defined as the (maximum) probability of rejecting at least one true null hypothesis, irrespective of the configuration of true and false null hypotheses 17.

To control the FWER, we use the closure principle (CP) 18. The CP considers all intersection hypotheses that are constructed from the elementary null hypotheses. To strongly control the FWER, an elementary null hypothesis H can only be rejected if all intersection hypotheses implying H are rejected also. For more details, we refer the reader to the papers of Bretz et al. 13, 16. In order to test an intersection hypothesis at the end of a two‐stage trial, we begin by correcting for multiplicity for each stage separately. Only afterwards do we combine the resulting adjusted p‐values into a pre‐specified combination function C(p,q). Given an intersection hypothesis H (where I⊆{1,…,K}) and the corresponding multiplicity‐adjusted stage 1 p‐value p 1 and stage 2 p‐value p 2, we reject H in the final analysis if (where c is a suitably chosen critical value to ensure a pre‐specified type I error rate of α).

As an example, consider using the closed testing procedure for the stage 1 data with early stopping for futility, using the Bonferonni correction for multiplicity (for the sake of simplicity). The usual first‐stage (unadjusted) p‐values for treatment i∈{1,…,K}, denoted p 1,, are as follows: For notational convenience, let denote the standardised treatment difference for treatment i∈{1,…,K}.

Consider comparing K = 3 treatments with a control (as in our example), as shown in Figure 3.

Figure 3

Closed testing procedure for the stage 1 data using the Bonferonni correction, with K = 3 treatments.

By the CP, treatment 1 (say) continues to stage 2 if Without loss of generality, suppose r(X 1) > r(X 2) > r(X 3). Then maxi∈{1,2,3}r(X ) = r(X 1), and treatment 1 continues to stage 2 if r(X 1) > Φ −1(1 − α 0/3). Hence, conditional on the event Q={:r(X 1) > r(X 2) > r(X 3),r(X 1) > Φ −1(1 − α 0/3)}, the UMVCUE for θ 1=μ 1−μ 0 is given by Equation (3), where K = 3 and b = Φ −1(1 − α 0/3).

If treatment 2 also continues to stage 2, then the UMVCUE for θ 2=μ 2−μ 0 is given by Equation (6), where we set K = 3, b 1=Φ −1(1 − α 0/3), b 2=Φ −1(1 − α 0/2) and b 3=−∞. Finally, if treatment 3 continues to stage 2, then the UMVCUE for θ 3=μ 3−μ 0 is given by Equation (6), with K = 3, b 1=Φ −1(1 − α 0/3), b 2=Φ −1(1 − α 0/2) and b 3=Φ −1(1 − α 0).

Example analysis

Returning to the data from our example trial, we can calculate the stage 1 Bonferroni‐adjusted p‐values as given as follows: Because all of the adjusted p‐values are less than α 0, the futility boundary threshold is not crossed for any of the doses, and hence, all of the dose groups (and placebo) are continued to stage 2. Plugging in the observed values (and known variances), the naïve estimator, stage 2 estimator and UMVCUE for the differences between the doses are given in Table 3. We see that the UMVCUE can be higher or lower than both the naïve and stage 2 estimators and is not necessarily closer to the stage 2 data.

Table 3

Estimators for the treatment differences from a seamless phase II/III trial.

Stage 1 Rank	Treatment	Naïve	Stage 2	UMVCUE
1	3	2.505	2.200	2.285
2	2	2.250	2.500	2.020
3	1	1.900	2.000	2.062

UMVCUE, uniformly minimum variance conditionally unbiased estimator.

As a brief comparison, the Kimani estimator for the highest ranked treatment is 2.197 using the modified formula shown in Section 3.2. Both these values are lower than the the UMVCUE and the stage 2 estimator, which may be a reflection of the fact that Kimani estimator overcorrects for bias. Note that the Kimani estimator is only for the highest ranked treatment, and estimates for the other treatment differences are unavailable.

Finally, if we want to test the elementary hypotheses H 1,H 2 and H 3 at the end of the trial, we first need to find the stage 2 Bonferroni‐adjusted p‐values: In order to combine the adjusted p‐values, we use the well‐known weighted inverse normal combination function C(p,q) = 1−Φ[w 1 Φ −1(1 − p) + w 2 Φ −1(1 − q)], where the weights w 1,w 2 are set proportional to the originally planned stage‐wise sample sizes: . Because α 0=0.1, then setting α = 0.025 means that the critical value c = 0.0401.

Taking dose level 3 as an example, by the CP, to reject H 3, we also need to reject the intersection hypotheses H {1,2,3},H {1,3} and H {2,3}. Because C(p 1,{1,2,3},p 2,{1,2,3}) < c, C(p 1,{1,3},p 2,{1,3}) < c, C(p 1,{2,3},p 2,{2,3}) < c and C(p 1,3,p 2,3) < c, then we can indeed reject H 3 and conclude that treatment 3 is superior to placebo. Similarly, following the same procedure, we can also reject H 2 and H 1.

Note that unlike hypothesis testing for the closure principle that strongly controls the type I error regardless of the number of treatments that continue to stage 2, if the estimates obtained after stage 2 are used to select the most effective treatment, all the estimators compared in Table 3 are biased. That is, the estimators derived in this section are only unbiased if the treatments will not be ranked after stage 2. This is because the estimators do not adjust for additional selection at the end of the trial.

Discussion

In seamless phase II/III trials, it may be desirable to explicitly take into account differences in the precision of the treatment effect estimates. Also, if more than one treatment is taken forward to the second stage, then it is natural to estimate the effect of treatments other than the highest ranked. In this paper, we described a framework for unbiased estimation that is applicable in full generality for the jth‐ranked treatment, where the precision of treatment effect estimates can take arbitrary values. Our generalised early stopping rules for futility means that our methodology can be applied where the interim selection rules are driven by formal hypothesis testing procedures, as would be expecting in practice.

Our UMVCUE for the maximum treatment difference is different analytically from the Kimani et al. estimator in the special case where the treatment effect variances are equal within each stage. Somewhat counter‐intuitively, our numerical simulations showed that when this special case is satisfied, our UMVCUE is slightly less efficient. The reason is that the Kimani estimator uses all of the data X 0,X 1,…,X ,Y 0,Y 1 explicitly, whereas the UMVCUE uses only the treatment differences X 1−X 0,…,X −X 0,Y 1−Y 0. Hence, if selection is indeed based on the observed stage 1 sample means, then we would recommend using the Kimani estimator (or its modification when variances in different treatment arms cannot be assumed equal).

When we do in fact have unequal variances and rank by standardised treatment difference, then our simulation results demonstrate how the Kimani estimator overcorrects for bias, because it conditions on different selection rules from those being actually used. This negative bias can be particularly severe when the ratio of the stage 1 treatment variances is greater than 1:2, and in these cases, the MSE of the Kimani estimator increases above the UMVCUE. These results indicate that the difference between the selection rules (and hence the estimators) is greatest when there is reason to believe that the treatment effect variances are different from treatment to treatment, such as when different classes of drugs are being tested and we have variance estimates from pilot studies.

Hence, if selection is based the standardised treatment differences, then we would recommend using the new UMVCUE, because it is unbiased and generally has a lower MSE. Note that our new estimator complements the existing Kimani estimator (and its modification). Indeed, the two frameworks are answering different questions because of the different selection rules being used.

We also showed how to extend our framework to estimate the jth‐ranked treatment effect for j > 1, in contrast to the Kimani estimator, which is only for the largest treatment difference. Our extended UMVCUE can then be applied within the context of formal hypothesis testing, where we correct for multiplicity with the closure principle. For simplicity, we used the Bonferroni correction in our example, but our framework could also be extended to work with more powerful multiplicity adjustment methods, such as the Simes, Holm or Hochberg procedures (as described in, e.g. 19).

In this paper, we only looked at point estimation of the treatment difference. However, in practice, it is natural to also seek confidence intervals at the end of the study. One possibility would be to use a parametric bootstrap procedure, similar to that described in 20 and 21. Alternatively, it might be possible to adapt the analytic approach of Sampson and Sill 22 to the seamless phase II/III trial setting.

A limitation of our work is that we assume that the variance of the treatment differences are known. In practice, if we have individual variance estimates for each treatment arm, then these will be less precise than a pooled estimate. In order to correctly account for this, one avenue of research is to extend the formulae of Cohen and Sackrowitz 8, who derived the UMVCUE in the independent normal setting (but without the option of early stopping) where the variances are unknown and have to be estimated.

A possible extension is to consider trials where there is early stopping for efficacy. This would be especially compatible with much of the literature on the combination test approach, where there can be early rejection of the null hypothesis 16. However, we anticipate that the UMVCUE would become much more complex in this setting, because of the additional restrictions on the support of Y . Finally, it is an open question whether there exist UMVCUEs for the treatment differences that are functions of all of the data (,) instead of just (,Y 0,Y ). If such estimators do exist, then they may outperform both our UMVCUE and the Kimani estimator in terms of MSE.