Point estimation following two-stage adaptive threshold enrichment clinical trials.

Recently, several study designs incorporating treatment effect assessment in biomarker-based subpopulations have been proposed. Most statistical methodologies for such designs focus on the control of type I error rate and power. In this paper, we have developed point estimators for clinical trials that use the two-stage adaptive enrichment threshold design. The design consists of two stages, where in stage 1, patients are recruited in the full population. Stage 1 outcome data are then used to perform interim analysis to decide whether the trial continues to stage 2 with the full population or a subpopulation. The subpopulation is defined based on one of the candidate threshold values of a numerical predictive biomarker. To estimate treatment effect in the selected subpopulation, we have derived unbiased estimators, shrinkage estimators, and estimators that estimate bias and subtract it from the naive estimate. We have recommended one of the unbiased estimators. However, since none of the estimators dominated in all simulation scenarios based on both bias and mean squared error, an alternative strategy would be to use a hybrid estimator where the estimator used depends on the subpopulation selected. This would require a simulation study of plausible scenarios before the trial.

INTRODUCTION

An area of recent interest in development of new therapies is stratified medicine, which involves using a biomarker to stratify patients into subgroups to distinguish those with the best likelihood of responding to particular treatments. If a biomarker has two levels, it is common to refer to one level as biomarker negative and the other as biomarker positive. We consider predictive biomarkers that allow the possibility of differences in treatment effects in different subpopulations, that is, a treatment by biomarker interaction effect.1

Advances in genetics have played a key role in stratified medicine, where biomarkers are based on genes. This has led to targeted therapies, where investigators determine a target subset of patients (subpopulation) and develop a drug (a targeted therapy) expected to be more efficacious than the control for these patients, and is possibly not beneficial to others. The target subpopulation may consist of patients with a certain gene (specifically a gene containing a certain allele) or platform of genes (specifically certain alleles corresponding to multiple genes). However, genes are not the only characteristics that are used to define a subset of patients. Examples of other biomarkers in the cancer setting include the size of tumor, protein level in the blood, and graded scores. When the clinical utility of the biomarker is not very strong or clear from previous studies, the biomarker stratified design may be used to test the effect of an experimental treatment. In this design, a trial enrolls patients from the full population but with provision for analyses of outcomes from the subpopulation.

One methodological challenge in stratified medicine is how to design and analyze efficient clinical trials that incorporate identification of the subpopulation that will benefit from the experimental treatment. An efficient design in late phase clinical trials is the two‐stage adaptive enrichment design.2 In stage 1, patients are recruited from the full population and data are used to perform an interim analysis to decide whether, in stage 2, enrollment will be from the full population or the subpopulation. The final confirmatory analysis uses data from both stages. Although the design is efficient because stage 1 data are used for subpopulation selection and confirmatory analysis, the latter is complex because of inclusion of subpopulation selection data.

We consider the case of a continuous (or a graded score) biomarker where the cut‐off value to distinguish between biomarker positive and negative patients is not definite from previous trials. Consequently, several candidate cut‐off values are possible, with trial data used to determine the cut‐off value. Simon and Simon2 refer to such a design that includes threshold determination as an adaptive threshold enrichment design. We give examples of clinical trials where this design can be used in Section 2.1.

Subpopulation selection based on the treatment effect can be advantageous because using an appropriate rule, the subgroup is selected in the case where there is apparent benefit in the subgroup and not in its complement (qualitative interaction) such as was observed by Mok et al.3 The full population is selected if there is apparent benefit in the full population including when the drug benefits the subgroup and its complement with different magnitudes (quantitative interaction) such as was observed in Tran et al.4 A subpopulation selection based on a hypothesis test for interaction only would not be able to distinguish between the two types of interactions.

Previous research that considers analysis of adaptive threshold enrichment trials focuses on control of type I error rate and power with less emphases on point estimation.2, 5 Recently, Li et al6 have derived expressions for the biases of estimators that ignore the adaptation but do not propose point estimators that account for subpopulation selection. Kimani et al7 and Kunzmann et al8 have developed estimators for a setting analogous to a single fixed cut‐off value. However, these estimators do not allow for using stage 1 data to determine the cut‐off value in an adaptive threshold enrichment trial.

A setting similar to an adaptive threshold enrichment design is that of treatment selection, where a control is compared to multiple experimental treatments, with stage 1 data used to select the experimental treatment to test further in stage 2.9, 10, 11, 12, 13, 14, 15, 16 Although several point estimators for this setting exist, they cannot be applied directly in adaptive threshold enrichment clinical trials because the correlation structure of the stage 1 sample means used for selection is different.

In this paper, we develop estimators that account for subpopulation selection following adaptive threshold enrichment trials using the principles that have been used to obtain point estimators that account for treatment selection. Two unbiased estimators build on the works by Kimani et al7 and Robertson et al.17 Two estimators build on the works by Whitehead18 and Stallard and Todd10 and involve deriving the bias function to calculate bias and subtracting bias from the naive estimator. The last is a shrinkage estimator and builds on the works by Hwang19 and Carreras and Brannath.14

DESCRIPTION OF THE SETTING AND NAIVE ESTIMATION

Motivation and notation

A condition where continuous biomarkers are tested and so the adaptive threshold design may be used is depression. Examples of continuous predictive biomarkers in depression are protein levels in the blood and an electrophysiological measure.20 While introducing notation, we describe features of clinical trials that are key in our methodology based on the setting of depression.

Patients' outcomes will be assumed to be normally distributed with a known standard deviation σ. In the context of depression, Uher et al20 perform simulations to give a guidance of the treatment effect size to be sought when predictive biomarkers are evaluated. One outcome measure they consider that is widely used in trials is the Hamilton Rating Scale for Depression (HRSD) score and is usually assumed to be normally distributed. For a trial of a prespecified duration of treatment, the aim may be to estimate the mean difference (experimental arm minus control arm) in HRSD scores between two interventions at the final follow‐up visit. Based on two trials,21, 22 the standard deviation of HRSD scores may be taken to be 7, that is, σ = 7.

We will consider trials that allow stopping for futility at an interim analysis if the observed treatment difference is less than some value b that we refer to as the futility boundary. The UK NICE guidelines recommend that an intervention for depression should demonstrate a difference of at least 3 HRSD points20 to be considered superior to its comparator. Therefore, at an interim analysis, the treatment may be deemed not to warrant further testing if the observed mean difference <2 (slightly less than the recommended value of 3), that is, b = 2.

We assume that a single continuous biomarker is used to identify the patients who benefit from a new intervention. We assume that in regard to biomarker values, there is monotonicity in treatment effect so that a higher biomarker value leads to a bigger treatment effect or a higher biomarker value leads to a smaller treatment effect. For ease of notation, we use the latter to develop methodology. Note that, if a higher biomarker value leads to a bigger treatment effect, the biomarker values can be transformed by multiplying by −1.

Using some biomarker threshold values, the full population (F) is partitioned into distinct partitions. For example, if F is subdivided into four partitions, the candidate threshold values c 1, c 2, c 3, and c 4 are such that patients in partitions 1, 2, 3, and 4 have biomarker values less than c 1, between c 1 and c 2, between c 2 and c 3, and between c 3 and c 4, respectively. The true mean differences in partitions 1 to 4 are denoted by δ 1, δ 2, δ 3, and δ 4, respectively. We denote the number of partitions by K so that, in this case, K = 4. We refer to the parts of F below threshold values c 1, c 2, c 3, and c 4 as subpopulations S 1, S 2, S 3, and S 4. Note for K = 4, S  = S 4 = F, and S 1, S 2, S 3, and S 4 consist of partition 1, partitions 1 and 2, partitions 1 to 3, and partitions 1 to 4, respectively. The true mean differences in S 1, S 2, S 3, and S 4 are denoted by θ 1, θ 2, θ 3, and θ 4, respectively. If, as expected, a higher biomarker value leads to a smaller treatment effect, then δ 1 ≥ δ 2 ≥ δ 3 ≥ δ 4 and θ 1 ≥ θ 2 ≥ θ 3 ≥ θ 4.

We assume that the threshold values c 1,…,c are prespecified. There are different ways for the choice of the thresholds values. For K = 4, quartiles may be used so that the prevalences for S 1 to S 4 are p 1 = 0.25, p 2 = 0.50, p 3 = 0.75, and p 4 = 1, respectively. Consequently, the partitions have equal prevalence (0.25) since if we set p 0 = 0, p  − p  = 0.25(i = 1,…,4). In some instances, the threshold values are chosen based on aspects such as biological activity so that the prevalences for partitions are not equal. Figure 1 summarizes the partitioning of F for any K ≥ 3.

Figure 1

Partitioning of the full population. Partitions to the left are expected to have bigger treatment effects. The pairs in the brackets are true mean differences and prevalences for partitions and candidate subpopulations

Hypothetical two‐stage adaptive threshold enrichment clinical trial

Predictive assessment of continuous biomarkers can been done in single‐stage clinical trials.23, 24 The alternative is to use the two‐stage adaptive threshold enrichment design, which is more efficient as more resources can be focused on the subpopulation that is most likely to benefit from the new treatment.24 The design has been used in recent trials with time‐to‐event (progression‐free survival) outcome data.5, 25, 26 As we propose in this paper, the design can be similarly used in trials with normally distributed outcome data. We note in Section 6 that the methods developed in this paper can be adapted for time‐to‐event outcome data.

We describe the form of the adaptive threshold enrichment design that we consider based on a hypothetical trial for depression, where for example protein level is used to partition F into quartiles. In stage 1, the trial recruits n 11 = 90, n 12 = 90, n 13 = 90, and n 1 = n 14 = 90 patients in partitions 1 to 4. The number of patients in S 1 to S 4 are m 11 = 90, m 12 = 180, m 13 = 270, and m 1 = n 14 = 360, respectively, since (i = 1,…,4). For simplicity, we assume that, in each partition, the 90 patients are equally split between the control and the experimental treatment. The outcome of interest is HRSD score and is assumed to be normally distributed with σ = 7. Let , , , and , the stage 1 sample mean differences in partitions 1 to 4 are , , , and , respectively. Let , , , and , the stage 1 sample means in S 1 to S 4 are , , , and , respectively. If the number of patients in a partition is not equally split between the control and the experimental treatment, the expressions for to and to are different. Note that, in this hypothetical trial, , , , and . The random vectors and have a linear relationship and are multivariate normal with mean vectors =(δ 1,δ 2,δ 3,δ 4)′ and =(θ 1,θ 2,θ 3,θ 4)′, respectively (see supplementary material). Hence, selection rules based on observed values for can be restated using the observed values for and vice versa.

Since a higher biomarker value is expected to lead to lower treatment effect, the largest subpopulation for which the observed stage 1 sample mean difference (in HRSD scores) is ≥b is selected to continue to stage 2. If the observed stage 1 sample mean differences in S 1, S 2, S 3, and S 4 = F are all less than b, the trial stops for futility. Note that the selected subpopulation is a random variable determined by observed stage 1 data. We use lower case s as the index for the “observed” selected subpopulation, with S denoting the selected subpopulation. At the end of stage 2, the primary objective is to obtain an estimate for θ , using an estimator that has good properties such as being mean unbiased and having small mean squared error (MSE).

Suppose that the stage 1 observed sample mean differences in partitions 1 to 4 are , , , and so that S 1 to S 4 stage 1 observed sample mean differences are , , , and . Subpopulation 2 would be selected, that is, S  = S 2, since it is the largest subpopulation with observed mean difference of at least 2 points, so that θ  = θ 2.

In stage 2, the trial recruits n 21 = 120 and n 22 = 120 patients in partitions 1 and 2, respectively. The number of patients in S 1 and S 2 are m 21 = 120 and m 22 = 240, respectively, since (i = 1,…,s). The sample sizes n 21 and n 22 and, hence, m 21 and m 22, should be prespecified in advance for example by fixing the total stage 2 sample size and the ratio of allocation to the selected partitions. Let and , the stage 2 sample mean differences in partitions 1 and 2 are and , respectively. Let and , the stage 2 sample means in S 1 and S 2 are and , respectively. For this hypothetical trial, and . Table 1 summarizes the notation we have introduced for any K ≥ 3. When a subscript in a notation includes two indices, the first corresponds to stage and the second to partition or subpopulation.

Table 1

Summary of notation

		Stage 1 Partitions					Stage 2 Partitions
Measure	Subgroup	1	2	…	K − 1	K	1	…	s  ∈  {1,…,K }
Upper threshold		c 1	c 2	…	c K − 1	c K	c 1	…	c s
Sample size	Partition	n 11	n 12	…	n 1,K − 1	n 1K	n 21	…	n 2S
	Subpopulation	m 11	m 12	…	m 1,K − 1	m 1K	m 21	…	m 2s
Sample variance	Partition	τ112	τ122	…	τ1,K−12	τ1K2	τ212	…	τ2s2
	Subpopulation	σ112	σ122	…	σ1,K−12	σ1K2	σ212	…	σ2s2
True mean	Partition	δ 1	δ 2	…	δ K − 1	δ K	δ 1	…	δ s
	Subpopulation	θ 1	θ 2	…	θ K − 1	θ K	θ 1	…	θ s
Sample mean	Partition	X¯_11	X¯_12	…	X¯_1,K−1	X¯_1K	X¯_21	…	X¯_2s
	Subpopulation	Y¯_11	Y¯_12	…	Y¯_1,K−1	Y¯_1K	Y¯_21	…	Y¯_2s

Suppose that, in stage 2, the observed sample mean differences in partitions 1 and 2 are and . Consequently, the stage 2 observed sample mean difference for S 2 is . The naive estimate for θ 2 is the two‐stage sample mean difference for S 2 given by . We describe in Section 2.4 that the naive estimates are biased because they ignore subpopulation selection. The aim of this paper is to develop estimators that adjust for subpopulation selection. The estimators are based on the selection rule described for the hypothetical trial, which we state for any K ≥ 3 partitions in the next section, and are conditional on the observed ordering of stage 1 data.

Selection rule

We derive estimators that are unbiased or with small bias conditional on the following specific selection rule. Other selection rules are considered in the discussion. Let b denote a futility boundary. The trial stops after stage 1 if for all i(i = 1,…,K). The trial continues to stage 2 with the full population (S ) if and with subpopulation S  ∈ {1,…,K − 1} if and for all i ∈ {s + 1,…,K}. Thus, as shown in the supplementary material, subpopulation S is selected if , where Equivalently, subpopulation S is selected if for all i ′ ∈ {1,…,s}, , where and

Naive estimation

For the selected subpopulation S , define t  = m 1/(m 1 + m 2). The naive estimator for θ that ignores subpopulation selection is

This is biased because the first term in (1) includes data used in the selection. Let and Prob(S ) denote the indicator and probability of selecting S , respectively. The conditional bias is

Using the joint density for or to compute Prob(S ) and is computationally time consuming because the limits of integration for each element in the vector depend on the values of the other elements. To overcome this, we use Z = (Z 1,…,Z )′, where and (i ′ = 2,…,K). The density for Z and the expressions for Prob(S ) and are provided in the supplementary material.

ESTIMATORS THAT ACCOUNT FOR SUBPOPULATION SELECTION

Unbiased estimators

General principles of obtaining unbiased estimators

One technique to account for subpopulation selection is Rao‐Blackwellization. By the Rao‐Blackwell theorem, conditional on a sufficient and complete statistic based on stages 1 and 2 data, the expected value of a conditionally unbiased estimator from the stage 2 data is the uniformly minimum variance conditional unbiased estimator (UMVCUE). We consider two methods for obtaining unbiased estimators for θ : deriving an UMVCUE for θ directly or, because the relationship between and is linear, deriving the UMVCUE for each δ (i = 1,…,s) and using a linear function to obtain an unbiased (though not necessarily minimum variance) estimator for θ . The latter builds on the work by Kimani et al.7 The former would involve correlated stage 1 statistics in the vector and builds on the work by Robertson et al.17

Uniformly minimum variance unbiased estimator following the work of Robertson et al (2016a)

The UMVCUE for θ is the expected value of conditional on a sufficient and complete statistic. As before, let denote the naive estimator for θ given by expression (1) and U be as u in Section 2.3 with replaced with . Following the work of Robertson et al,17 the UMVCUE for θ is where , , and φ(.) and Φ(.) denote the density and distribution functions of a standard normal, respectively.

Unbiased estimator following the work of Kimani et al (2015)

The UMVCUE for (i ′ = 1,…,s) is the expected value of conditional on a sufficient and complete statistic. Let (i ′ = 1,…,s) denote the naive estimator for . Furthermore, let and be as and in Section 2.3 with replaced with . Following the work of Kimani et al,7 the UMVCUE for (i ′ = 1,…,s) is where and . Consequently, the unbiased estimator for θ is

Bias‐adjusted estimators

An overview of bias‐adjusted estimation

Another technique to account for subpopulation selection would be to utilize the fact that we can calculate bias of the naive estimate using expression (2). The naive estimate is then adjusted by subtracting the bias. However, expression (2) is a function of (or equivalently ), the vector of the unknown treatment effects. To overcome this, we estimate bias, and hence, bias‐adjusted estimators obtained in this way are not necessarily mean unbiased.

Single‐iteration bias‐adjusted estimator

We consider two bias‐adjusted estimators. For the first one, the bias is estimated based on the observed sample mean differences (i = 1,…,K). Let and denote the bias estimator for θ obtained by replacing with in expression (2) to get an adjusted estimator for θ of We will refer to this estimator as the single‐iteration bias‐adjusted estimator.

Multiple‐iteration bias‐adjusted estimator

For the second bias‐adjusted estimator, the bias is estimated iteratively.10, 13, 18 Let (i = 1,…,K) denote the naive estimator for θ and . The biases for the naive estimators depend on and we denote bias for (i = 1,…,K) by b () and the vector by (). The second adjusted estimator, which we refer to as multiple‐iteration bias‐adjusted estimator is obtained by solving iteratively. Using similar notation, alternatively, one could solve and then use the relationship between and to obtain a bias‐adjusted estimate for θ . For the simulations in Section 5, we solve and with an accuracy of 0.001, convergence was achieved in almost all simulated trials. Suppose that the solution is obtained at iteration and let denote the bias for when is taken to be , then the multiple‐iteration adjusted estimate for δ is and the multiple‐iteration bias‐adjusted estimator for θ is

The details of calculating are given in the supplementary materials.

Shrinkage estimators

General principles for shrinkage estimation

A third technique for accounting for subpopulation selection is to use shrinkage methods. Hwang19 considered the case of estimating a treatment mean after ordering independent sample means in a single‐stage trial for K ≥ 4. A subpopulation selection rule that corresponds to Hwang's case is that of selecting only one partition based on some ordering of . We initially consider Hwang's selection rule and denote the selected partition by s . Hwang assigns a common normal prior distribution N(μ,ν 2) to each δ (i = 1,…,K). The posterior mean for , its Bayes estimator, is , where C = 1 − 2σ 2/(2σ 2 + n ν 2) and n is stage 1 sample size in each intervention in each partition. Replacing the unknown μ and C with their unbiased estimators and , respectively, gives the empirical Bayes estimator. Let , Hwang indicates that a better estimator, which we refer to as the shrinkage estimator, is .

Carreras and Brannath14 extended the work to two‐stage trials. Define to be the proportion of stage 1 data. The two‐stage shrinkage estimator for is . For K < 4, Carreras and Brannath propose defining . Using the fact that the estimator of Hwang19 applies for all parameters δ (i = 1,…,K) and that its examination by Carreras and Brannath showed that it works for any rule used to pick the parameters on which to make inference, in Sections 3.3.2 and 3.3.3, we extend this work to give two shrinkage estimators for the subpopulation selection rule in Section 2.3.

First shrinkage estimator

As in unbiased estimation, we consider both combining shrinkage estimators for treatment effects in partitions to obtain an estimator for θ and directly obtaining a shrinkage estimator for θ . From Section 3.3.1, the shrinkage estimator for δ (i = 1,…,s) is , where and for K ≥ 4, , whereas for K < 4, . The first shrinkage estimator for θ is

Second shrinkage estimator

The second shrinkage estimator, which we denote by , involves using the entire parameter vector . A multivariate normal prior for is specified and updated with the data . The resulting posterior is multivariate normal with nonzero covariance, and hence, the iterative procedure of Morris27 and Brüncker et al28 is utilized to obtain (see supplementary material).

WORKED EXAMPLE

We use data from the hypothetical trial for depression in Section 2.2 to demonstrate how to compute the naive ( ), the UMVCUE ( ), the unbiased ( ), the single‐iteration bias‐adjusted ( ), the multiple‐iteration bias‐adjusted ( ), the first shrinkage ( ), and the second shrinkage ( ) estimates. We also use the example to demonstrate differences among the various estimates in a single trial. The data and the various estimates are summarized in Table 2. The explicit computations for the various estimates and the R program used are provided in the supplementary material. Here, we only give explicit details of computing and as they are easier to compute, and since based on the simulations in the next section, we recommend .

Table 2

Worked example data and estimates

Data and Summary Measures
		Stage 1 Partitions				Stage 2 Partitions		Estimating θ 2
Measure	Subgroup	1	2	3	4	1	s = 2	Estimator	Estimate
Sample	Partition	n 11 = 90	n 12 = 90	n 13 = 90	n 14 = 90	n 21 = 120	n 22 = 120	θ^_2,N	2.614
size	Subgroup	m 11 = 90	m 12 = 180	m 13 = 270	m 14 = 360	m 21 = 120	m 22 = 240	θ^_2,UMV	2.839
Sample	Partition	τ112=2.178	τ122=2.178	τ132=2.178	τ142=2.178	τ212=1.633	τ222=1.633	θ^_2,U	2.965
variance	Subgroup	σ112=2.178	σ122=1.089	σ132=0.726	σ142=0.544	σ212=1.633	σ222=0.817	θ^_2,SI	2.633
Sample	Partition	x¯_11=3	x¯_12=2	x¯_13=0.8	x¯_14=0	x¯_21=3	x¯_22=2.4	θ^_2,MI	2.666
mean	Subgroup	y¯_11=3	y¯_12=2.5	y¯_13=1.93	y¯_14=1.45	y¯_21=3	y¯_22=2.7	θ^_2,L_1	2.164
								θ^_2,L_2	2.194

For the UMVCUE ( ) given by expression (3), the first term . Furthermore, and so that and . Since ( p  − p ) = 0.25 for all i = 1,…,4, then u (the observed value for U) is given by Note that f (b) = 1.268 × (2.614 − 2) = 0.779 and f (u) = 1.268 × (2.614 − 2.6) = 0.018, so that substituting into expression (3), .

For the unbiased estimator ( ) given by expression (4), we make the following calculations. The naive estimates for partitions 1 and 2 are and , respectively. Note that and . Since p  − p  = 0.25(i = 1,…,4), = 4 × [(0.5 × 0.2) − (0.25 × 2)] = 2 and , respectively. For partition 1,

Similarly, for partition 2, w 2 = 2.2. Then, for partition 1, f (v 1) = 0.896 × (3 − 2) = 0.896 and f (w 1) = 0.896 × (3 − 3.2) = −0.179, and for partition 2, f (w 2) = 0.896 × (2.229 − 1) = 1.101 and f (w 2) = 0.896 × (2.229 − 2.2) = 0.026. Now, we have all components required to obtain UMVCUEs for the effects in partitions 1 and 2, which give and , respectively. The unbiased estimate is the weighted sum of the UMVCUEs in the partitions giving .

The estimates , , , and are greater than (see Table 2). This may be explained by the observation in Section 5.2 that, in some scenarios, the naive estimator is negatively biased. The estimate is slightly smaller than . Again, this may be explained by an observation in Section 5.2 that, for all scenarios in the simulation study, on average, the single‐iteration bias‐adjusted estimator gives a smaller estimate than the multiple‐iteration estimator.

SIMULATIONS TO COMPARE THE VARIOUS ESTIMATORS

Simulations setting

To evaluate the properties of the various estimators, we conducted simulations with σ 2 = 1 and b = 0. We initially consider the case of K = 4 and p  − p  = 0.25(i = 1,…,4). In all simulations, if the trial continues to stage 2, the combined stages 1 and 2 sample size is set to be 800. For example, if the stage 1 sample size is 400 patients, the stage 2 sample size is 400. The available patients in stage 1 are equally split among the four partitions and treatment arms. For example, with 400 patients in stage 1, in each partition, 50 patients are randomly allocated to each of the control and experimental treatment. Similarly, the patients available for testing in stage 2 are equally split among the partitions that continue to stage 2 and among the treatment arms. Hence, with 400 patients available in stage 2, if F is selected, the patient allocation in stage 2 is as in stage 1 with 400 patients. If S 2 is selected so that two partitions are tested in stage 2, in each partition, 100 patients are randomly allocated to each of the control and experimental treatment. We perform simulations for three cases of stage 1 sample size (200, 400, and 600 patients). Taking the combined stages 1 and 2 to be 800 patients is justified in the supplementary material.

We consider seven scenarios with true treatment effects as summarized in Table 3. The selection rule and estimators developed are aimed at identifying predictive effects, but since we are estimating mean differences, the methods are valid with or without prognostic effects. If the biomarker has no predictive effect but has a prognostic effect, we are in a scenario of equal treatment effects in all partitions. Scenarios 1, 3, and 7 could be such cases. If there are prognostic and predictive effects, we are in a scenario of unequal treatment effects in partitions. Scenarios 2, 4, 5, and 6 could be such cases. In Scenarios 1 to 3, the right decision is to continue to stage 2 with F, but with decreasing probability of selecting F. The right decisions for Scenarios 4 to 6 are to continue with S 3, S 2, and S 1, respectively. The ideal decision for Scenario 7 is to stop at stage 1. The probabilities for various decisions for different scenarios when stage 1 includes 200 patients (25 in each treatment arm in each partition) are also given in Table 3. These have been calculated using expressions in Section 2.4 and in the supplementary material. As expected, the probability of stopping the trial at stage 1 (last column) increases as the treatment effects in partitions become less than b in more partitions (from 0.007 for Scenario 1 to 0.482 for Scenario 7). In each of Scenarios 4 to 6, the probability of continuing with F is substantially larger than the probability of making the right decision, demonstrating that, in some configurations, decision making is challenging. In Section 5.2.1, simulations show that incorrect decisions tend to be made when observed means are substantially different from the true means and hence lead to bias.

Table 3

Treatment effects and probabilities of different decisions for the various scenarios in the simulation study (probabilities of correct decisions are in bold)

	Treatment Effect					Probability of a Decision (n 1 = 200)
Scenario	δ 1	δ 2	δ 3	δ 4	Ideal Selection	F	S 3	S 2	S 1	Stop
1	0.3	0.3	0.3	0.3	F	0.983	0.005	0.003	0.002	0.007
2	0.2	0.1	0.1	0.1	F	0.812	0.049	0.035	0.034	0.070
3	0.0	0.0	0.0	0.0	F	0.500	0.083	0.070	0.073	0.274
4	0.1	0.0	0.0	−0.2	S 3	0.430	0.179	0.093	0.093	0.205
5	0.1	0.0	−0.2	−0.1	S 2	0.362	0.112	0.179	0.115	0.232
6	0.1	−0.2	−0.1	−0.1	S 1	0.298	0.098	0.104	0.214	0.286
7	−0.1	−0.1	−0.1	−0.1	Stop	0.240	0.083	0.087	0.108	0.482

Table 4 gives probabilities of various decisions when the stage 1 sample sizes are 400 and 600. For scenario 3, where treatment effects are equal in all partitions and equal to the futility boundary, the probabilities of various decisions are approximately equal for different stage 1 sample sizes. For the other scenarios, by comparing the probabilities in bold, the probability of making a correct decision increases with stage 1 sample size.

Table 4

Probabilities of different decisions for different stage 1 sample sizes for various scenarios in the simulation study (probabilities of correct decisions are in bold)

	Ideal	Probability of a Decision (n 1  =  400)					Probability of a Decision (n 1  =  600)
Scenario	Selection	F	S 3	S 2	S 1	Stop	F	S 3	S 2	S 1	Stop
1	F	0.9987	0.0004	0.0002	0.0002	0.0005	0.99988	0.00004	0.00002	0.00002	0.00004
2	F	0.8944	0.0312	0.0212	0.0200	0.0332	0.93711	0.02033	0.01326	0.01213	0.01717
3	F	0.5000	0.0833	0.0698	0.0734	0.2735	0.50000	0.08333	0.06981	0.07342	0.27344
4	S 3	0.4013	0.2286	0.0983	0.0971	0.1747	0.37973	0.26859	0.10095	0.09838	0.15235
5	S 2	0.3085	0.1220	0.2386	0.1261	0.2048	0.27015	0.12853	0.28802	0.13147	0.18183
6	S 1	0.2266	0.0977	0.1156	0.2939	0.2662	0.17916	0.09454	0.12250	0.35893	0.24487
7	Stop	0.1587	0.0724	0.0842	0.1157	0.5690	0.11034	0.06193	0.07895	0.11756	0.63122

For each of the seven scenarios and three different stage 1 sample sizes, we simulated stage 1 data for N = 1 000 000 trials. For each trial, the subpopulation with the largest simulated sample mean difference ≥0 continues to stage 2. If no subpopulation fulfills this, the trial stops. We consider estimation conditional on continuing to stage 2 and so bias and MSE for each estimator are evaluated based on simulated trials that continue to stage 2. Using for illustration, for each s , bias and MSE are calculated as and .

Simulation results

Comparing biases for the various estimators

Figure 2 summarizes biases when the stage 1 sample size is 200 (top plots) and 600 patients (bottom plots). Plots for the case where the stage 1 sample size is 400 patients are provided in the supplementary material. Plots in Columns 1 to 4 correspond to the cases of selecting F, S 3, S 2, and S 1, respectively. The y‐axes correspond to biases divided by approximate standard errors (SEs). The approximate and so SEs are only equal when F is selected (Column 1). Although SEs are not equal, we will later observe from the boxplots of the estimates that the trend for bias is the same when bias is not divided by SE. The x‐axes correspond to the seven scenarios. As per the legend, biases for different estimators are distinguished by different line types. Estimators and are not included in Figure 2 because they are mean unbiased. For Scenario 1, the probabilities for selecting S 3, S 2, and S 1 are low and so simulations results are highly variable when S 3, S 2, or S 1 is selected but this does not change the general findings in this paper.

Figure 2

Biases in units of approximate standard error for different configurations. The dotted line is the point of no bias. Other line types correspond to different estimators. SE, standard error

We first describe the results for the case where the stage 1 sample size is 200 (top row). When F is selected, the naive estimator ( ) and the first shrinkage estimator ( ) are the same and correspond to the line showing the largest biases. Focusing on the naive estimator, the bias when F is selected (Column 1) is positive in all scenarios. For scenarios where the right decision is to continue with F (Scenarios 1 to 3, see Table 3), bias when F is selected is attributable to the futility rule with the bias negligible when the effect in F is substantially larger than the futility boundary (Scenario 1). When the right decision is not to continue with F (Scenarios 4 to 7) but F is selected, the impact of selection and futility on bias would increase and consequently give a larger bias. Still focusing on the top row, when S 3 is selected (Column 2), the naive estimator for θ 3 is negatively biased for some scenarios and positively biased for other scenarios. The explanation for this pattern is given in the supplementary material. Comparing the bias when F, S 3, S 2, and S 1 are selected (Columns 1 to 4), the bias is smallest when S 1 is selected. This can be attributed partly to the enrichment, where the stage 2 sample size is fixed regardless of the size of the population selected so that when S 1 is selected, proportionally, there are more unbiased stage 2 data to estimate θ 1 compared to when F, S 3, or S 2 is selected. In summary, note that, in some scenarios, the bias of the naive estimator is substantial and so it is essential to use an estimator that corrects for subpopulation selection.

Still focusing on the top row, when F is selected, practically, the single‐iteration bias corrected estimator is mean unbiased, especially for Scenarios 1 to 3 where the correct decision is to select F. When S 3 is selected, almost eradicates bias in Scenarios 3 to 7 and is better than the naive estimator in Scenarios 1 and 2. When S 2 or S 1 is selected, eradicates almost all bias in Scenarios 2 to 7 but does not do so in Scenario 1. In all scenarios, the line for the multiple‐iteration bias‐adjusted estimator ( ) is always slightly above that of . Hence, comparing and , when is negatively biased, is preferable, whereas is preferable when it is positively biased.

Comparing biases for the naive estimator for different stage 1 sample sizes (top versus bottom plots), as also indicated by expression (2), the bias increases with the proportion of stage 1 data. Increase in bias is also seen for both the single‐iteration ( ) and multiple‐iteration ( ) bias‐adjusted estimators. From the bottom row, and perform worst when some partitions that should be dropped at stage 1 continue to stage 2 or when some partitions that should continue to stage 2 are dropped. As before, the line for is above that of with the distances between the lines increasing with stage 1 sample size.

The pattern of the shrinkage estimators is best understood by considering all results in Figure 2. In all cases, the line for the first shrinkage estimator ( ) overlaps or is above that of the second shrinkage estimator ( ). Estimator performs similar to or better than when the selected subpopulation consists of partitions that should continue to stage 2 such as when F is selected in Scenarios 1 to 3 and such as when S 3 is selected in Scenarios 1 to 4. Estimator performs better than when the selected subpopulation consists of partitions that should not continue to stage 2 such as when F is selected in Scenarios 4 to 7 and such as when S 3 is selected in Scenarios 5 to 7.

In almost all scenarios, the two shrinkage estimators perform worse than the other estimators that account for adaptation. One reason for this may be the fact that the shrinkage estimators do not account for stopping for futility. When F is selected, the naive estimator is the same as the first shrinkage estimator. This is because the stage 1 estimate in partition i is so that the shrinkage estimator shrinks to the effect in the full population, that is, to . A reasonable alternative would be to use a weighted mean of , , …, . For example, if we shrink to , in terms of sample means in partitions, we are shrinking to a weighted sum such that for i < i ′, has more weight than . In such a case, shrinkage estimators will be closer to the naive estimators when fewer partitions are selected (see additional simulations in the supplementary material).

Comparing MSEs for the various estimators

Mean squared errors for the various estimators are given in Figure 3. The y‐axes are root mean squares ( ) divided by approximate SEs. The best shrinkage estimator in terms of bias (either or depending on the scenario) has smaller or practically the same MSEs as the naive estimator. Hence, the best shrinkage estimators may be considered to be better than the naive estimator in terms of MSE. The challenge, however, is determining the best shrinkage estimator since the true treatment means are unknown.

Figure 3

Root mean squares in units of approximate standard error for different configurations. Different line types correspond to different estimators. MSE, mean squared error; SE, standard error

Since estimators that extend the works of Kimani et al ( ) and of Robertson et al ( ) are mean unbiased, their MSEs are variances. When S 1 is selected, by derivation, the two estimators are the same and, hence, have equal MSE. For any other selection, as expected, has smaller MSE than . The differences increase with stage 1 sample size (top versus bottom plots) and the size of the selected subpopulation (right to left panels). The MSEs of and are mostly larger than the MSEs for all the other estimators with the differences substantial when selection is performed later in the trial.

In general, the MSEs for the single‐iteration ( ) and multiple‐iteration ( ) bias‐adjusted estimators are practically the same. Hence, since their biases are also similar, the two estimators are approximately equivalent and so it is sufficient to compare one of them to the other estimators. The MSE for is larger than that of the naive estimator ( ) in most cases while it is always smaller than the MSEs for the unbiased estimators ( and ).

Comparing the estimators using both bias and MSE

Comparing the shrinkage estimators ( and ) to the naive estimator ( ), we prefer . This is because although a shrinkage estimator sometimes has a smaller MSE, it can have substantially higher bias than (for example, compare Columns 4 in Figures 2 and 3).

Comparing the single‐iteration bias‐adjusted estimator ( ) and the naive estimator ( ), when F is selected, is preferable as it reduces bias substantially and has smaller MSE. However, when S 1 is selected, is better as it has smaller MSE and it does not differ from in terms of bias. When S 3 or S 2 is selected, is better when bias is not substantial (Scenarios 3 and 4), whereas for Scenarios 5 to 7, is better as it reduces bias and its MSE is better or only slightly higher than that of . Overall, we consider as a better estimator than as it performs better in cases with substantial bias.

When F is selected, the bias of the naive estimator ( ) is substantial and compared to the UMVCUE ( ), we prefer the latter since the difference in RMSE between the two estimators is smaller than the bias eradicated. When S 1 is selected, we would also recommend over as the former is mean unbiased in all scenarios, with the only case where it is not clearly superior due to high RMSE being when n 1 = 600. The conclusion when S 3 or S 2 is selected is the same as when S 1 is selected, that is, is better than .

Comparing the single‐iteration bias‐adjusted estimator to the UMVCUE , we recommend the latter since, when F is selected, has substantial bias that is larger than the difference in RMSE between it and . In addition, when S 1 is selected, the difference in RMSE between the two estimators is smaller than the bias of . Consequently, based on the performance across the scenarios in the simulation study, we recommend when an adaptive threshold enrichment design is used.

For a more detailed comparison of the estimators, Figures 4 and 5 give boxplots of simulated estimates for Scenarios 1 (top plots), 4 (middle plots), and 6 (bottom plots) described in Table 3 when F and S 3 are selected. The boxplots emphasize the findings summarized above. As an example, when n 1 = 600(Figure 5), for Scenario 6 (bottom left panel), almost all naive estimates are above the true value and performs well in that case. From the left panels, we note the unbiased estimators ( and ) have substantially higher variances compared to the other estimators.

Figure 4

Boxplots of estimates for different estimators when n 1 = 200. Results have been chosen when F and S 3 were selected and for Scenarios 1 (top panels), 4 (middle panels), and 6 (bottom panels). The dashed lines correspond to the true means in the selected subpopulation

Figure 5

Boxplots of estimates for different estimators when n 1 = 600. Results have been chosen when F and S 3 were selected and for Scenario 1 (top panels), 4 (middle panels), and 6 (bottom panels). The dashed lines correspond to the true means in the selected subpopulation

Summary findings and recommendations from the simulation study

The bias of the naive estimator can be substantial, and so it is essential to use an estimator that corrects for the decision made using stage 1 data. We recommend the estimator that follows the work of Robertson et al ( ) since it is mean unbiased. Although it has larger MSE than some estimators, the bias eradicated in most cases was larger than the difference in RMSEs. Although the simulation study was based on four partitions and specific treatment effect scenarios, we expect similar findings for other configurations (that is, more candidate partitions and/or different effect sizes). Simulations for the case of 8 partitions are in the supplementary material.

We have recommended one estimator for all scenarios. An alternative is a hybrid estimator where the recommended estimator ( or ) depends on the subpopulation selected. This is suitable if investigators are willing to sacrifice unbiasedness for more precision. In this case, before the trial, a simulation study based on plausible scenarios would be required to compare bias and MSE conditional on the selected subpopulation.

DISCUSSION

Acknowledging that different patients may require different care has led to trial designs that incorporate assessment of treatment effects in different subsets of the population. Most statistical methodologies for such designs focus on hypothesis testing.2, 5, 24, 26, 29, 30, 31, 32, 33 In this paper, we have considered point estimation following an adaptive threshold enrichment clinical trial. We have assessed bias for the naive estimator when different subpopulations are selected. Depending on the scenario, the bias of the naive estimator of the treatment effect in the selected subpopulation is substantial and can be negative or positive. There is thus a need for new estimators. Building on estimators that have been proposed for treatment selection, we have derived several estimators that account for subpopulation selection. By derivation, two estimators are mean unbiased. In this paper, we have recommended the best among these two, that is, the UMVCUE. An alternative is a hybrid estimator where different estimators are recommended based on the selected subpopulation. This would require a simulation study before the trial and is suitable if investigators can accept some unbiasedness for a more precise estimator.

We have considered a specific selection rule but the proposed estimators can be modified for other selection rules. For example, it may be desired that different subpopulations have different futility boundaries. Futility boundaries may be based on factors such as subpopulation prevalence, and sponsor and public health gains.34 Another factor is safety where the futility boundary may be chosen to reflect investigators' willingness to accept moderate efficacy if the new treatment is substantially safer than the control. The selection rule we have used specifies that a higher biomarker value leads to a smaller treatment effect. If this is a misspecification of the relationship between the biomarker and treatment effect, the unbiased estimators will remain so because we condition on the selection rule. However, the probability of making the right decision will be low and we anticipate that the naive estimator will have more bias and that the unbiased estimators will have higher MSE.

In the derivations, we have not required the prevalences in different partitions to be equal. If the biomarker values are approximately continuous, then it is reasonable to subdivide the full population into equal partitions as we have done in the example and the simulations. Other numerical biomarker values may be discrete with few possible values, leading to partitions with varying sizes.

We have assumed the number of patients in each partition, and hence prevalence, is known. For the case of two partitions and a fixed cut‐off value, taking the stage 1 number of patients in a partition to have a binomial distribution, Kimani et al7 showed that using stage 1 prevalence estimates in the expressions for the unbiased estimators provides unbiased estimates for the treatment effects. This extends to the case of more than two partitions, where numbers of patients in partitions are taken to have a multinomial distribution. The proof is based on the fact that the estimator in a partition is unbiased conditional on the number of patients in an interval and that the proportion of patients in a partition is unbiased for the prevalence in the partition. The proof for the case of estimating the cut‐off values using stage 1 data is similar.

Conditional on continuing to stage 2, we have derived estimators for the effect in the selected subpopulation. Continuing to stage 2 is necessary for the unbiased estimators. This is not the case for the other estimators as they involve obtaining stage 1 estimates in all partitions that correct for the subpopulation selection and then combine them with the stage 2 unbiased estimates. Hence, estimates for effects in the dropped partitions that correct for subpopulation selection can be obtained using the shrinkage and bias‐adjusted estimators. However, they are not necessarily mean unbiased.

Methods developed for normally distributed data following treatment selection have been adapted for time‐to‐event data.28 Even after assuming asymptotic normality of the log hazard ratio, some of the estimators we have derived such as the UMVCUE may not be valid for time‐to‐event data. For example, if there is a quantitative interaction with hazard ratios in different partitions being unequal, a model that accounts for this is required. In this case, obtaining separate estimates for each partition is the valid approach.

Finally, since in all simulations, the combined stages 1 and 2 sample size was 800, for the different stage 1 sample sizes considered, there would be no savings or losses in terms of the cost of treating patients. The saving/loss is only made in terms of costs associated with biomarker testing. Hence, the case for performing subpopulation selection with a small proportion of patients can be justified if the biomarker is expensive, leading to savings if F is selected. The case for performing subpopulation selection with a large proportion of patients is justifiable if the biomarker is not expensive. In this case, the resources loss is not substantial if F is selected and yet, if only a part of the population will benefit, there is a higher probability of making the right decision that may improve power. The setting of fixed total sample size is sometimes referred to as enrichment because if some partitions are dropped in stage 2, the number of patients recruited from partitions in stage 2 is higher than if more partitions were selected. To save money on treatment costs or reduce the total sample size, subpopulation selection could be performed early, with no enrichment in stage 2. With no enrichment, the number of patients in a partition in stage 2 is fixed. The statistical properties of the estimators for the setting with no enrichment can be evaluated as in the case of enrichment.

SIM7831‐sup‐0001‐supplementary information.pdf

Click here for additional data file.