Supplementation of a clinical trial by historical control data: is the prospect of dynamic borrowing an illusion?

There are strong arguments, ethical, logistical and financial, for supplementing the evidence from a new clinical trial using data from previous trials with similar control treatments. There is a consensus that historical information should be down-weighted or discounted relative to information from the new trial, but the determination of the appropriate degree of discounting is a major difficulty. The degree of discounting can be represented by a bias parameter with specified variance, but a comparison between the historical and new data gives only a poor estimate of this variance. Hence, if no strong assumption is made concerning its value (i.e. if 'dynamic borrowing' is practiced), there may be little or no gain from using the historical data, in either frequentist terms (type I error rate and power) or Bayesian terms (posterior distribution of the treatment effect). It is therefore best to compare the consequences of a range of assumptions. This paper presents a clear, simple graphical tool for doing so on the basis of the mean square error, and illustrates its use with historical data from clinical trials in amyotrophic lateral sclerosis. This approach makes it clear that different assumptions can lead to very different conclusions. External information can sometimes provide strong additional guidance, but different stakeholders may still make very different judgements concerning the appropriate degree of discounting.