Interpretation of Results from Under-accruing Studies.

A phase II study reported in this issue of The Oncologist demonstrates potential evidence of benefit in progression‐free survival (PFS) with the use of metformin in addition to carboplatin, paclitaxel, and bevacizumab in patients with chemotherapy‐naïve or metastatic nonsquamous non‐small cell lung cancer (NSCLC). As often happens with many clinical trials, this study was stopped early because of slow accrual, with nearly 60% fewer patients than had been originally planned; however, the study result was statistically significant, thereby prompting questions about how to interpret the results in light of the trial's early stopping and under‐accrual.

This trial reported by Marrone et al. [1] randomized patients with chemotherapy‐naïve advanced nonsquamous NSCLC to combination therapy of carboplatin, paclitaxel, and bevacizumab with or without metformin and concluded that metformin has a potential role in cancer treatment based on results obtained from a cohort of 25 patients. The primary endpoint was 1‐year PFS; the study was originally designed to randomize 45 patients to the experimental arm with 84% power to detect an increase in the 1‐year PFS to 30%, from a null 1‐year PFS of 15% based on historical data from an ECOG 4599 study [2]. Fifteen patients were to be randomized to the control arm for confirmation of those historical control data.

The study stopped early because of slow accrual associated with increasingly frequent use of pemetrexed as first‐line and/or maintenance treatment of patients with metastatic nonsquamous NSCLC at the institution. On this trial, a total of 18 patients were randomized to and received carboplatin, paclitaxel, and bevacizumab with metformin followed by bevacizumab and metformin (Arm A), and 6 patients were randomized to control and therefore received carboplatin, paclitaxel, and bevacizumab followed by bevacizumab maintenance alone (Arm B). (One patient from Arm A was taken off study after the first cycle of treatment because of a reaction to paclitaxel and was therefore removed from the primary efficacy analysis.)

The study found that the addition of metformin improved PFS at 1 year, compared with the historical benchmark. The 1‐year PFS on Arm A was 47% (95% confidence interval [CI], 25%–88%), and metformin was deemed promising because the 95% lower confidence bound exceeded 15%, the hypothesized 1‐year PFS under the null. We note that the primary hypothesis test relies solely on the single‐arm comparison of the experimental arm (Arm A) to historical control and does not make a direct comparison with the randomized control arm (Arm B) in this trial, which was included for confirmation of the historical control estimates. We also note that caution should be taken when confirming the historical control with the control arm (Arm B), because it is not practical to think that results from six patients could serve to validate results from the prior randomized phase III trial. The median PFS associated with the metformin arm was 9.6 months (95% CI, 7.3 months, not reached (NR)), compared with 6.7 months (95% CI, 4.4 months, NR) for the control arm, resulting in a log‐rank p value of .024. Although the log‐rank test here was statistically significant, technically it was not the primary test according to the statistical design of the trial and therefore should not be used to draw the primary conclusion of the study.

In general, there is a variety of reasons why a clinical trial can stop early, including but not limited to efficacy, futility, safety concerns, or slow accrual. Although this trial was stopped early because of slow accrual, the authors obtained a statistically significant result, albeit with approximately 40% of the intended sample size. The main question that arises is whether or not the positive result can be interpreted as scientifically interesting or meaningful, particularly in the context of other available and potentially more effective therapies such as pemetrexed or immunotherapy. It would be important to consider the precision of the estimates of the efficacy endpoints when interpreting the results of the trial, because the sample size affects the variability of those estimates. This can be done by studying the confidence interval about the point estimate for the primary endpoint. For example, the two‐sided 95% confidence interval about the 1‐year PFS rate on this study was 25%–88%, indicating a fairly wide range of values with which the primary result is consistent: combination therapy with metformin could confer a 1‐year PFS as high as 88% or as low as 25%. After considering the magnitude of benefit of other available therapies, the lower and upper bounds of this confidence interval can help to indicate how meaningful the PFS result from this trial may be to the scientific community, particularly with respect to ruling out an unacceptably low 1‐year PFS estimate. Therefore, in the same way the confidence interval was used to rule out the 15% historical estimate from the prior study of carboplatin, paclitaxel, and bevacizumab, the same can be done using estimates of other available regimens [2]. For example, the 1‐year PFS in the study of pembrolizumab by Reck at al. [3] was approximately 50%; the estimated lower bound of confidence interval from the metformin trial does not appear superior to this. Therefore, some might conclude that it may not be reasonable to continue investigating metformin for patients with this disease.

Another issue with interpreting results of trials that stopped early is the decrease in power because of the reduced sample size, which is predefined as the minimum number of patients required to have a reasonable chance of detecting of an expected therapeutic effect. This trial was planned such that 45 patients randomized to the metformin arm would have provided 84% power to detect the stated treatment effect. It would have been reasonable for the authors to state what the power of the study for that same treatment effect was with the reduced sample size.

Other considerations to be placed into the context of the interpretation of such a trial include definitions of the endpoints. For example, on this trial patients were censored at the time they completed or were removed from study treatment rather than continuing to be followed until progression or death. It has been shown by Campigotto and Weller [4] that this type of informative censoring can lead to biased estimates of the time‐to‐event endpoint, in which the degree of bias largely depends on the proportion of patients who were informatively censored. Although not the case on the trial by Marrone and colleagues, the definition of progression could also have an effect on the results and interpretation, particularly if investigator‐assessed or modified response criteria were used. The context of response rates can become challenging because patients who die or progress prior to their first disease evaluation are often removed from the “evaluable” population, despite RECIST guidance [5]. For interpretation of overall survival, it would be important to know how treatment was defined on the study, because allowing treatment beyond progression would affect this endpoint.

It is far more common for under‐accruing trials to report “negative” results, or results that are not statistically significant. Much of what has been stated above in the context of this “positive” trial applies to these trials as well; however, we note a key point about the interpretation of those trials: failure to reject the null hypothesis and report a significant finding does not mean that a treatment effect does not exist or that the regimens studied are equal to one another. It could simply mean that the study was underpowered to demonstrate the therapeutic effect of interest.

Stopping studies early is unfortunately not a rare occurrence in oncology, particularly for slow accrual. In our experience, there is often a perceived lack of value in reporting these trials, but from a statistical perspective peer‐reviewed publication should be encouraged so that the results can be made more widely available. It should be acknowledged that Marrone et al. have conducted and appropriately reported the study under discussion and hopefully helped to inform the allocation of additional patient and clinical resources for future research endeavors.