Replication of Randomized, Controlled Trials Using Real-World Data: What Could Go Wrong?

With the growing interest in using real-world evidence (RWE) for regulatory purposes, researchers and policy makers are considering how best to assess the credibility of RWE. Because the randomized controlled trial (RCT) has long been regarded as the gold standard for high-quality research, one approach being pursued is to see to what extent findings from RCTs can be replicated based on analyses of nonrandomized real-world data (RWD). If findings are congruent, the reasoning goes, this would bolster confidence in the underlying RWD sources and validity of the RWE generated. But it is well known that medical interventions perform differently in experimental clinical trials versus real-world clinical practice, reflecting a phenomenon known as the "efficacy-effectiveness gap." So even with the highest-quality RWD sources and strongest analytic methods, we can and should expect to observe discrepancies in findings between RCTs and RWE. This calls into question the objectives of RCT replication efforts and makes clear that impugning RWD sources and analytic methods for failing to align with RCT findings is inappropriate and, worse, potentially harmful to the growing acceptance of RWE in stakeholder decision making.